2024-11-12 15:35:37 [scrapy.utils.log] INFO: Versions: lxml 5.2.1.0, libxml2 2.12.6, cssselect 1.2.0, parsel 1.9.1, w3lib 2.2.1, Twisted 24.10.0, Python 3.10.13 (main, Mar 22 2024, 12:17:55) [GCC 11.4.0], pyOpenSSL 24.2.1 (OpenSSL 3.3.1 4 Jun 2024), cryptography 43.0.0, Platform Linux-5.15.0-122-generic-x86_64-with-glibc2.35
2024-11-12 15:35:37 [dh] ERROR: File not found: [Errno 2] No such file or directory: '/root/Sourava/varta/crawler/dh_scrapy/spiders/Data/test.json'
2024-11-12 15:35:37 [scrapy.addons] INFO: Enabled addons:
[]
2024-11-12 15:35:37 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2024-11-12 15:35:37 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'dh_scrapy',
'FEED_EXPORT_ENCODING': 'utf-8',
'LOG_LEVEL': 'INFO',
'NEWSPIDER_MODULE': 'dh_scrapy.spiders',
'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['dh_scrapy.spiders'],
'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2024-11-12 15:35:37 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2024-11-12 15:35:37 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-11-12 15:35:37 [scrapy.middleware] INFO: Enabled item pipelines:
['dh_scrapy.pipelines.DhScrapyPipeline']
2024-11-12 15:35:37 [scrapy.core.engine] INFO: Spider opened
2024-11-12 15:35:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-11-12 15:35:37 [scrapy.core.engine] INFO: Closing spider (finished)
2024-11-12 15:35:37 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'elapsed_time_seconds': 0.003567,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2024, 11, 12, 10, 5, 37, 312277, tzinfo=datetime.timezone.utc),
'log_count/INFO': 8,
'memusage/max': 64516096,
'memusage/startup': 64516096,
'start_time': datetime.datetime(2024, 11, 12, 10, 5, 37, 308710, tzinfo=datetime.timezone.utc)}
2024-11-12 15:35:37 [scrapy.core.engine] INFO: Spider closed (finished)
2024-11-12 15:35:37 [scrapy.utils.log] INFO: Versions: lxml 5.2.1.0, libxml2 2.12.6, cssselect 1.2.0, parsel 1.9.1, w3lib 2.2.1, Twisted 24.10.0, Python 3.10.13 (main, Mar 22 2024, 12:17:55) [GCC 11.4.0], pyOpenSSL 24.2.1 (OpenSSL 3.3.1 4 Jun 2024), cryptography 43.0.0, Platform Linux-5.15.0-122-generic-x86_64-with-glibc2.35
2024-11-12 15:35:37 [dh] ERROR: File not found: [Errno 2] No such file or directory: '/root/Sourava/varta/crawler/dh_scrapy/spiders/Data/test.json'
2024-11-12 15:35:37 [scrapy.addons] INFO: Enabled addons:
[]
2024-11-12 15:35:37 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2024-11-12 15:35:37 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'dh_scrapy',
'FEED_EXPORT_ENCODING': 'utf-8',
'LOG_LEVEL': 'INFO',
'NEWSPIDER_MODULE': 'dh_scrapy.spiders',
'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['dh_scrapy.spiders'],
'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2024-11-12 15:35:37 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2024-11-12 15:35:37 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-11-12 15:35:37 [scrapy.middleware] INFO: Enabled item pipelines:
['dh_scrapy.pipelines.DhScrapyPipeline']
2024-11-12 15:35:37 [scrapy.core.engine] INFO: Spider opened
2024-11-12 15:35:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-11-12 15:35:37 [scrapy.core.engine] INFO: Closing spider (finished)
2024-11-12 15:35:37 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'elapsed_time_seconds': 0.003567,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2024, 11, 12, 10, 5, 37, 312277, tzinfo=datetime.timezone.utc),
'log_count/INFO': 8,
'memusage/max': 64516096,
'memusage/startup': 64516096,
'start_time': datetime.datetime(2024, 11, 12, 10, 5, 37, 308710, tzinfo=datetime.timezone.utc)}
2024-11-12 15:35:37 [scrapy.core.engine] INFO: Spider closed (finished)