ogdch/s3-logs-analyzer
Watch
2
mirror of https://github.com/ogdch/s3-logs-analyzer.git synced 2026-08-25 22:14:17 +00:00
Code Releases Activity
Generate daily download stats for files hosted on S3 from the logs | Read-only mirror of https://github.com/ogdch/s3-logs-analyzer — Bundesamt für Statistik. Issues & pull requests at the source.
9 commits 2 branches 2 tags 41 KiB
  • Python 100%
Find a file
Exact
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Stefan Oderbolz 3add511ee7 Merge pull request #3 from aguereca/master 
Fix strict cert issue on Python 2.7.9 plus added support of bucket names
2015-08-04 09:07:16 +02:00
.gitignore initial version which downloads, parses and lists the downloads by amount in a .txt file 2013-09-27 16:23:37 +02:00
LICENSE.md Add MIT license 2015-06-03 22:25:24 +02:00
parse.py update README, use .csv as extension 2013-11-06 10:57:53 +01:00
README.md update README, use .csv as extension 2013-11-06 10:57:53 +01:00
requirements.txt initial version which downloads, parses and lists the downloads by amount in a .txt file 2013-09-27 16:23:37 +02:00
sync.py Use default values on boto get_bucket function 2015-08-03 17:07:50 -07:00

README.md

S3-Logs-Analyzer

  • Download the access logs and parse them to retrieve the actual access counts for the files being hosted on S3
  • Generate a CSV with fields: file, date, downloads

Setup

$ virtualenv pyenv --no-site-packages
$ source pyenv/bin/activate
$ pip install -r requirements.txt

Download S3 log files (only downloads non existing ones)

$ python sync.py [the flags]
$ get a coffee (if initial_import)

Parse

$ python parse.py --folder [the folder name where the log files got downloaded to]