OwlCyberSecurity - MANAGER
Edit File: accesslog.pyc
� �t�ac @ s� d Z d d d d g Z d d l Z d d l Z d d l Z d d l Z y d d l Z Wn e k rq e Z n Xd d l m Z d d l Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z m Z d d l m Z d d l m Z e � Z d e f d � � YZ d e f d � � YZ d e f d � � YZ! d e" f d � � YZ# d e j$ f d � � YZ% d e f d � � YZ& d e j' f d � � YZ( d e j) f d � � YZ* d S( s Access logs. t DirectoryLogWrapperResourcet! RotatingFileAccessLoggingObservert AMPCommonAccessLoggingObservert AMPLoggingFactoryi����N( t platform( t getAdjustedMethodName( t Logger( t RecordType( t iweb( t BaseCommonAccessLoggingObserver( t LogWrapperResource( t protocolt task( t amp( t configc B s e Z d � Z d � Z RS( c C s# t t | � j | � | | _ d S( N( t superR t __init__t directory( t selft resourceR ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR : s c C s | j S( N( R ( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt getDirectory? s ( t __name__t __module__R R ( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR 8 s t% CommonAccessLoggingObserverExtensionsc B s e Z d Z d � Z RS( sR A base class for our extension to the L{BaseCommonAccessLoggingObserver} c C s d } d } | j d � t j k r�| d } | d } | d } d } t | d d � d k r� d � } | | j � } | | j � } | | k r� d j d | d | � } q� | } n t j r� t | d � r� d | j | j f } n | j } d } i | j j d 6| d 6| j | j j d d � � d 6| d 6| j j d d � d 6d j d � | j D� � d 6| j d 6| j d 6| j j d d � d 6| j j d d � d 6} t j rB| d 7} t j r�t j n d | d <| j r| d! 7} | j j j j | d" <n t j � } t j r�| j d d# } d$ t j � f | j d <xR | j D]G \ } } | d% | | f 7} | | d& | | <| d$ k rN| } qNqNWt | j � d# k r�| d' 7} | | d& | d( <q�n' | d) 7} | | j d d# d& | d$ <t | d* � r�x� t! | j" j# � d+ d, � �D]v \ } } t$ | � j d d � } t$ | � j d d � } d- | k r}d. | f } n | d/ | | f 7} | | | <q%Wn t | d0 � r�| d1 7} | j% j& � d# | d2 <n | j j' d3 d4 � } | rBd5 j g | D] } | d6 ^ q�� } | j d- d4 � } | d7 7} | | d8 <qBn | d d9 k rp| j j' d3 d4 � } | r�| d: j( d5 � d: j) � | d <| d; 7} d< | d= <q�qpn� d> | k rp| j d> � } d? } i | j* j+ d 6d d 6| j t j � � d 6d@ d 6d4 d 6d4 d 6dA d 6d d 6d d 6d d 6} t j rp| dB 7} | j* j, j- | dC <| d! 7} | j | d" <qpn | d k rxo | j. � D]a \ } } t/ | t0 � s�q�n | j dD dE � } | j dF dG � } | j d dH � } | | | <q�WdI | dJ <| | dK <| j1 | � n d S(L Nt interfacet requestt responset loginfot -t authnUserc S sM | j j t j k r# | j j d Sd j d | j j d | j j d � Sd S( Ni s ({rtype}){name}t rtypet name( t recordt recordTypeR t usert shortNamest format( t principal( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt convertPrincipaltoShortNameU s s "{authn} as {authz}"t authnt authzt submethods %s(%s)s� %(host)s - %(uid)s [%(date)s] "%(method)s %(uri)s HTTP/%(protocolVersion)s" %(statusCode)s %(bytesSent)d "%(referer)s" "%(userAgent)s"t hostt uidt datei t methodt "s %22t urit .c s s | ] } t | � Vq d S( N( t str( t .0t x( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pys <genexpr>~ s t protocolVersiont statusCodet bytesSentt referers user-agentt userAgents i=%(serverInstance)st 0t serverInstances or=%(outstandingRequests)st outstandingRequestsi t ts %s=%%(%s).1fi� s t-log=%(t-log).1fs t-logs t=%(t).1ft extendedLogItemst keyc S s | d S( Ni ( ( R3 ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt <lambda>� t t s "%s"s %s=%%(%s)st errors err=%(err)st errs x-forwarded-forR@ t ,i2 s fwd=%(fwd)st fwds 0.0.0.0i����s unix=%(unix)st truet unixt overloadedse %(host)s - %(uid)s [%(date)s] "%(method)s" %(statusCode)s %(bytesSent)d "%(referer)s" "%(userAgent)s"s ???i� s p=%(serverPort)st serverPorts s \rs s \ns \"s access-logt types log-format(2 t Nonet getR t IRequestt getattrR t authzUserR$ R t EnableExtendedAccessLogt hasattrR- R) t remoteAddrR* t logDateStringt headerst getHeaderR/ t replacet joint clientprotot codeR6 t LogIDt chanRequestt channelt factoryR; t timet EnableExtendedTimingAccessLogt timeStampst lent sortedR= t iteritemsR1 RB t qnamet getRawHeaderst splitt stript transportt hostnamet servert portt itemst isinstancet basestringt logStats( R t eventDictR$ t formatArgsR R R R+ R& t uidnt uidzR- t nowtimet basetimet tagt timestampt kt vt fwdHeaderst hdrt forwardedForRH ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt emitH s� + & ! ( R R t __doc__R} ( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR C s c B s� e Z d Z d � Z e d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z RS( s� Class to do "apache" style access logging to a rotating log file. The log file is rotated after midnight each day. This class also currently handles the collection of system and log statistics. c C s: | | _ d | _ g | _ d | _ d | _ d | _ d S( N( t logpathRK t systemStatst statsByMinutet stats1mt stats5mt stats1h( R R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR � s c C sb | j � r) | r) | j � | j � n t | t � rJ | j d � } n | j j | d � d S( s Log a message to the file and possibly rotate if date has changed. @param message: C{str} for the message to log. @param allowrotate: C{True} if log rotate allowed, C{False} to log to current file without testing for rotation. s utf-8s N( t shouldRotatet flusht rotateRm t unicodet encodet ft write( R t messaget allowrotate( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt accessLog� s c C sD t t | � j � | j � | j d t j j � j � f � d S( sM Start logging. Open the log file and log an "open" message. s Log opened - server start: [%s].N( R R t startt _openR� t datetimet nowt ctime( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� s c C sf | j d t j j � j � f t � t t | � j � | j � | j d k rb | j j � n d S( sM Stop logging. Close the log file and log an "open" message. s Log closed - server stop: [%s].N( R� R� R� R� t FalseR R t stopt _closeR� RK ( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� s & c C s> t | j d d � | _ | j t j | j � d � | _ d S( s$ Open the log file. t ai i N( t openR R� t toDatet ost statt lastDate( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� " s c C s | j j � d S( s% Close the log file. N( R� t close( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� * s c C s | j j � d S( s% Flush the log file. N( R� R� ( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� 1 s c C s$ t j r | j � | j k St Sd S( sC Rotate when the date has changed since last write N( R t RotateAccessLogR� R� R� ( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� 8 s c G s t j | � d S( s Convert a unixtime to (year, month, day) localtime tuple, or return the current (year, month, day) localtime tuple. This function primarily exists so you may overload it with gmtime, or some cruft to make unit testing possible. i ( R^ t localtime( R t args( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� B s c C sD y d j t t | � � SWn# d j t t | j | � � � SXd S( sP Return the suffix given a (year, month, day) tuple or unixtime t _N( RW t mapR1 R� ( R t tupledate( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt suffixN s c C s� d | j | j | j � f } t j j | � rH t j d d | �d S| j d t j j � j � f t � t j d d | d d �| j j � t j | j | � | j � | j d t j j � j � f t � d S( s� Rotate the file and create a new one. If it's not possible to open new logfile, this will fail silently, and continue logging to old logfile. s %s.%ss= Cannot rotate log file to '{path}' because it already exists.t pathNs Log closed - rotating: [%s].s Rotating log file to: '{path}'t systemt Loggings Log opened - rotated: [%s].( R R� R� R� R� t existst logt infoR� R� R� R� R� R� R� t renameR� ( R t newpath( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� Y s & c C s� t j j s t j j ru | j d k r6 t � | _ n d | k sR | d d k rV d S| j � } | j | | � n | d d k r� | j | d | � n d S( s Update stats RJ s access-logNs log-format( R t Statst EnableUnixStatsSockett EnableTCPStatsSocketR� RK t SystemMonitort ensureSequentialStatst updateStatsR� ( R t statst currentStats( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyRo l s c C s� t j j r t j j r i S| j d k r< t � | _ n | j � } | j d k r� t d t | j � � } | d k r� | j | d | _ q� | j � | _ n | j d k r | j � | _ t d t | j � � } x? t | d � D]* } | j | d } | j | j | � q� Wn | j d k r�| j � | _ t d t | j � � } x? t | d � D]* } | j | d } | j | j | � qdWn i | j j d 6| d 6| j d 6| j d 6| j d 6} | S( s" Return the stats i i i i i����i= R� t currentt 1mt 5mt 1hN( R R� R� R� R� RK R� R� R� t minRa R� t initStatsR� t ranget mergeStatsR� Rl ( R R� t indext iR� t printStats( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt getStats� s: c C s5 t t j � d � d } t | j � d k r� | j d d | k r� | j d d } | | k r� d | _ d | _ d | _ n x9 | | k r� | d 7} | j j | | j � f � q� Wq� n7 | j j | | j � f � d | _ d | _ d | _ d } t | j � | k r&| j | | _ n | j d d S( sK Make sure the list of timed stats is contiguous wrt time. g N@i< i i����iA i N( t intR^ Ra R� RK R� R� R� t appendR� ( R t dtindext oldindext threshold( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� � s& & c C s� d � } i d d 6t j t � d 6t j t � d 6d d 6d d 6d d 6d d 6d d 6d d 6| � d 6| � d 6d d 6| j j d d 6S( Nc S sC i d d 6d d 6d d 6d d 6d d 6d d 6d d 6d d 6d d 6S( Ni s <10mss 10ms<->100mss 100ms<->1ss 1s<->10ss 10s<->30ss 30s<->60ss >60ss Over 1ss Over 10s( ( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt initTimeHistogram� s i t requestsR- s method-tt 500t 401g R< s t-resp-wrt slotss max-slotst Ts T-RESP-WRs T-MAXs cpu uset cpu( t collectionst defaultdictR� t floatR� Rl ( R R� ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� � s c s� t | � } � d d k r) d � d <n � d c d 7<� d | c d 7<� d | c | j d d � 7<| d d k r� � d c d 7<n# | d d k r� � d c d 7<n � d c | j d d � 7<� d c | j d d � 7<� d c | j d d � 7<t � d t | d � r,| j j n d � � d <� d c | j j d 7<� f d � } | j d d � } | d k r�| | d � n t � d | � � d <| j d d � } | d k r�| | d � n d S( NR� i g R� i R- s method-tR< R5 i� R� i� R� s t-resp-wrR� R; s max-slotst limiters cpu usec s0 | d k r# � | d c d 7<n� | d k rF � | d c d 7<n� | d k ri � | d c d 7<n} | d k r� � | d c d 7<nZ | d k r� � | d c d 7<n7 | d k r� � | d c d 7<n � | d c d 7<| d k r � | d c d 7<n# | d k r,� | d c d 7<n d S( Ng L�@s >60si g L�@s 30s<->60sg ��@s 10s<->30sg @�@s 1s<->10sg Y@s 100ms<->1sg $@s 10ms<->100mss <10mss Over 1ss Over 10s( ( R< R> ( R� ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt histogramUpdate s" R� s T-MAXs T-RESP-WR( R RL t maxRQ R� t maxOutstandingRequestsR� Rl RK ( R R� R� t adjustedMethodR� R< ( ( R� s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� � s. 2 c s� � d d k r d � d <n � d c | d 7<x4 | d j � D]" } � d | c | d | 7<qB Wx4 | d j � D]" } � d | c | d | 7<qy W� d c | d 7<� d c | d 7<� d c | d 7<� d c | d 7<� d c | d 7<t � d | d � � d <� d c | d 7<� f d � } x4 | d j � D]" } � d | c | d | 7<qRWt � d | d � � d <x4 | d j � D]" } � d | c | d | 7<q�Wd S( NR� i g R� R- s method-tR� R� R< s t-resp-wrR� s max-slotsc s0 | d k r# � | d c d 7<n� | d k rF � | d c d 7<n� | d k ri � | d c d 7<n} | d k r� � | d c d 7<nZ | d k r� � | d c d 7<n7 | d k r� � | d c d 7<n � | d c d 7<| d k r � | d c d 7<n# | d k r,� | d c d 7<n d S( Ng L�@s >60si g L�@s 30s<->60sg ��@s 10s<->30sg @�@s 1s<->10sg Y@s 100ms<->1sg $@s 10ms<->100mss <10mss Over 1ss Over 10s( ( R< R> ( R� ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� 5 s" R� s T-MAXs T-RESP-WR( t keysR� ( R R� R� R- R� t bin( ( R� s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� $ s( ( R R R~ R t TrueR� R� R� R� R� R� R� R� R� R� Ro R� R� R� R� R� ( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR � s$ 1 # /R� c B s; e Z d Z e j d d � Z d � Z d � Z d � Z RS( s� Keeps track of system usage information. This installs a reactor task to run about once per second and track system use. t CPUStatst totalt idlec C s� i t d k r t j � n d d 6d d 6d d 6d d 6t j � d 6| _ t d k r� t j � } t j t | � | j � | _ n t j d d � | _ t j | j � | _ | j j d � d S( Ni����s cpu countg s cpu usei s memory useds memory percents start timeg �?( t psutilRK t cpu_countR^ Rl t cpu_timesR� R� t sumR� t previous_cpuR t LoopingCallt updateR� ( R t times( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR X s !c C s | j j � d S( s$ Just stop the task N( R R� ( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� j s c C s� t d k r� t j � } t j t | � | j � } y7 d d | j | j j | j | j j | j d <Wn t k r� d | j d <n X| | _ n t d k r� d t k r� t j � } | j | j d <| j | j d <n d S( Ng Y@g �?s cpu useg t freebsds memory useds memory percent( R� RK R� R� R� R� R� R� R� Rl t ZeroDivisionErrorR t virtual_memoryt usedt percent( R R� t cpu_nowt mem( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� p s 7 ( R� R� ( R R R~ R� t namedtupleR� R R� R� ( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� P s t LogStatsc B s e Z d e j � f g Z RS( R� ( R R R t Stringt arguments( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� � s c B s, e Z d � Z d � Z d � Z d � Z RS( c C s d | _ g | _ d S( N( RK R t _buffer( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR � s c C s1 | j r- x! | j D] } | j | � q Wn d S( N( R� Ro ( R t msg( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt flushBuffer� s c C s | | _ | j � d S( sG An AMP client connected; hook it up to this observer. N( R R� ( R t connectedClient( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt addClient� s c C s~ | j d k rj t j | � } t | t � r? | j d � } n | j j t d | �} | j t j � n | j j | � d S( s> Log server stats via the remote AMP Protocol s utf-8R� N( R RK t jsont dumpsRm R� R� t callRemoteR� t addErrbackR� RB R� R� ( R R� t d( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyRo � s ( R R R R� R� Ro ( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR � s t AMPLoggingProtocolc B s- e Z d Z d � Z d � Z e j e � RS( sC A server side protocol for logging to the given observer. c C s | | _ t t | � j � d S( N( t observerR R� R ( R R� ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR � s c C s# t j | � } | j j | � i S( N( R� t loadsR� Ro ( R R� R� ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyRo � s ( R R R~ R Ro R� t responder( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR� � s c B s, e Z d � Z d � Z d � Z d � Z RS( c C s | | _ d S( N( R� ( R R� ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR � s c C s | j j � d S( N( R� R� ( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt doStart� s c C s | j j � d S( N( R� R� ( R ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt doStop� s c C s t | j � S( N( R� R� ( R t addr( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt buildProtocol� s ( R R R R� R� R� ( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyR � s (+ R~ t __all__R� R� R� R� R� t ImportErrorRK t sysR R^ t calendarserver.logAnalysisR t twext.python.logR t twext.who.idirectoryR t txweb2R t txweb2.logR R t twisted.internetR R t twisted.protocolsR t twistedcaldav.configR R� R R R t objectR� t CommandR� R t AMPR� t ServerFactoryR ( ( ( s+ /opt/cpanel-ccs/calendarserver/accesslog.pyt <module> sB �� e3!