Code

668 joblogs#43

Code

Closed

vlerkin wants to merge 19 commits intoq-m:mainq-m/scrapyd-k8s:mainfrom vlerkin:668-joblogsvlerkin/scrapyd-k8s:668-joblogsCopy head branch name to clipboard

Commits

Commits on Nov 26, 2024

• remove limited reconnection attempts; cap backoff time with 15 min; configure logs in a separate file logging_config.py that is imported to the __main__.py to trigger logs configuration before othe…

  Show description for dfc939b
  vlerkin

  committed
  dfc939b

  Copy full SHA for dfc939b

• refactor code to make config import and logging configuration separate modules; implement logging to be configured globally but keep separate logger for each module; add level for logging to be con…

  Show description for f472580
  vlerkin

  committed
  f472580

  Copy full SHA for f472580

• change level to INFO in config instead of DEBUG to make it less confusing for users

  vlerkin

  committed
  19bfc19

  Copy full SHA for 19bfc19

• remove lines to configure the logging level

  vlerkin

  committed
  4761f26

  Copy full SHA for 4761f26

• add logging.getLevelName() method to be used instead of creating a dict with all available logging level to validate user input from the config file

  vlerkin

  committed
  8792063

  Copy full SHA for 8792063

• move logging configuration to the api.py; move config loading to the api.py; keep logging_config file

  vlerkin

  committed
  7609bb6

  Copy full SHA for 7609bb6

• rename logging_level to log_level

  vlerkin

  committed
  4e0405c

  Copy full SHA for 4e0405c

• remove redundant check of the log_level in the logging_config.py

  vlerkin

  committed
  82c55e2

  Copy full SHA for 82c55e2

• Small changes

  

  wvengen

  authored and

  vlerkin

  committed
  d47c9e8

  Copy full SHA for d47c9e8

• change timeout_seconds parameter for k8s api connection from 0 to 300 to reset the connection periodically; add lock for events like subscibe, unsubscribe and event dispatching

  vlerkin

  committed
  d274cac

  Copy full SHA for d274cac

Commits on Nov 28, 2024

• change timeout_seconds watcher param back to 0 to try and hold connection forever

  vlerkin

  committed
  f633d5a

  Copy full SHA for f633d5a

Commits on Dec 3, 2024

• remove job_name mapping and implement a stateless directory sreen to check for logfiles of the jobs that are still on the cluster in Completed or Failed state

  vlerkin

  committed
  9c64ce4

  Copy full SHA for 9c64ce4

Commits on Dec 4, 2024

• remove empty line

  vlerkin

  committed
  27317eb

  Copy full SHA for 27317eb

Commits on Dec 17, 2024

• make directory for logs configurable providing a default option; remove random component from log files names and keep job_name as one

  vlerkin

  committed
  a3de0b7

  Copy full SHA for a3de0b7

• describe a new parameter in CONFIG.md to provide info on how to set a directory for logs

  vlerkin

  committed
  fc50b64

  Copy full SHA for fc50b64

• remove job_name as it is not correct, use job_id for file names and pod_name to indicate which stdout (logs) to read from

  vlerkin

  committed
  1189e80

  Copy full SHA for 1189e80

• move logs_dir param to the joblogs section in config, adjust code to read it from there if provided

  vlerkin

  committed
  87ce2c8

  Copy full SHA for 87ce2c8

• remove default value from the code, make logs_dir a mandatory param in the joblogs section of the config file, in the KubernetesJobLogHandler init throw a value error if logs_dir is not provided in…

  Show description for b2dd992
  vlerkin

  committed
  b2dd992

  Copy full SHA for b2dd992

Commits on Dec 18, 2024

• added functionality to remove successfully ploaded files; additionaly on a check when old jobs are present on the cluster and the code checks that the files of those jobs already exist in the S3 st…

  Show description for b682edc
  vlerkin

  committed
  b682edc

  Copy full SHA for b682edc