Saving Scraped Data To Postgres Database With Scrapy Pipelines

If your scraping a website, you need to save that data somewhere. A great option is Postgres as it is a battletested database, trusted by thousands of companies to handle huge amounts of data.

In this guide, we will go through how to save our data to a Postgres database using Scrapy pipelines:

• What Are Scrapy Item Pipelines?
• Setting Up A Postgres Database
• Saving Data To A Postgre Database
• Only Saving New Data

First, let's go over what are Scrapy Item Pipelines.

webscan

webscan is a browser-based network IP scanner and local IP detector. It detects IPs bound to the user/victim by listening on an RTP data channel via WebRTC and looping back to the port across any live IPs, as well as discovering all live IP addresses on valid subnets by monitoring for immediate timeouts (TCP RST packets returned) from fetch() calls or hidden img tags pointed to valid subnets/IPs.

I2S Sound Tutorial for ESP32

In this tutorial you learn the fundamentals of the I2S communication that is used to transfer digital sound signals and why you should use an ESP32 microcontroller for your I2S projects.

After your learn the functionality of I2S we create 3 projects where you can apply what you have learned.

About Project Euler

What is Project Euler?

Project Euler is a series of challenging mathematical/computer programming problems that will require more than just mathematical insights to solve. Although mathematics will help you arrive at elegant and efficient methods, the use of a computer and programming skills will be required to solve most problems.

The motivation for starting Project Euler, and its continuation, is to provide a platform for the inquiring mind to delve into unfamiliar areas and learn new concepts in a fun and recreational context.

Who are the problems aimed at?

The intended audience include students for whom the basic curriculum is not feeding their hunger to learn, adults whose background was not primarily mathematics but had an interest in things mathematical, and professionals who want to keep their problem solving and mathematics on the cutting edge.

Currently we have 1314304 registered members who have solved at least one problem, representing 220 locations throughout the world, and collectively using 114 different programming languages to solve the problems.

Can anyone solve the problems?

The problems range in difficulty and for many the experience is inductive chain learning. That is, by solving one problem it will expose you to a new concept that allows you to undertake a previously inaccessible problem. So the determined participant will slowly but surely work his/her way through every problem.

"Project Euler exists to encourage, challenge, and develop the skills and enjoyment of anyone with an interest in the fascinating world of mathematics."

Circuit Simulator

This is an electronic circuit simulator. When the applet starts up you will see an animated schematic of a simple LRC circuit. The green color indicates positive voltage. The gray color indicates ground. A red color indicates negative voltage. The moving yellow dots indicate current.

To turn a switch on or off, just click on it. If you move the mouse over any component of the circuit, you will see a short description of that component and its current state in the lower right corner of the window. To modify a component, move the mouse over it, click the right mouse button (or control-click if you have a Mac) and select “Edit”.

A quoi ça sert… un pont diviseur de tension ?

Un pont de tension permet de diviser une tension d’entrée à l’aide de 2 résistances en série. Il permet ainsi d’obtenir une tension de sortie U2 plus faible que la tension d’entrée U. Il est utilisé couramment pour créer une tension de référence.

Flipper Zero

Multi-tool Device for Geeks

Flipper Zero is a portable multi-tool for pentesters and geeks in a toy-like body. It loves hacking digital stuff, such as radio protocols, access control systems, hardware, and more. It's fully open-source and customizable, so you can extend it in whatever way you like.

microdot

The impossibly small web framework for Python and MicroPython

Microdot is a minimalistic Python web framework inspired by Flask. Given its
small size, it can run on systems with limited resources such as
microcontrollers. Both standard Python (CPython) and MicroPython are supported.

from microdot import Microdot

app = Microdot()

@app.route('/')
async def index(request):
    return 'Hello, world!'

app.run()

Rapidly build interactive 3D apps for the web.

You already know Threlte

Threlte puts the simplicity of Svelte and all of the power of Three.js right at your fingertips. It's designed to be powerful and flexible while still being approachable and easy to use.

Reimagine the Web

With AR and VR picking up steam, the web is becoming more and more 3D. Threlte provides a simple, declarative API for creating 3D content on the web.

Powerful Integrations

Threlte comes with integrations for physics, animation and 3D model loading.

Sur l’ESP32 certaines broches sont reliées à l’ADC intégré du microcontrôleur. Un ADC est un convertisseur analogique vers numérique : il revoit une valeur numérique proportionnelle à la tension mesurée.

ESP8266-12E et ESP8266-07 ont une broche ADC facilement accessible. Cela signifie que ces cartes ESP8266 peuvent lire des signaux analogiques. Dans ce tutoriel, nous allons vous montrer comment utiliser la lecture analogique avec l’ESP8266 à l’aide du firmware Arduino IDE, MicroPython ou Lua.

La filière veut replacer la France et l’environnement au cœur de la fabrication de vélos. Pour encourager les professionnels et informer les consommateurs, elle lance le Cyclescore, basé sur des critères environnementaux et sociétaux, et l’indice France vélo, qui distinguera la fabrication française. Deux labels intéressants, qui devront toutefois trouver leur place sans créer de confusion dans l’esprit des consommateurs.

Man pages

• mosquitto - running the Mosquitto broker
• mosquitto.conf - the Mosquitto broker configuration file
• mosquitto_passwd - command line utility for generating Mosquitto password files
• mosquitto_pub - command line utility for publishing messages to a broker
• mosquitto_rr - command line utility for simple request/response with a broker
• mosquitto_sub - command line utility for subscribing to topics on a broker
• mosquitto-tls - brief cheat sheet for creating x509 certificates
• mqtt - description of MQTT features

teleinfo2mqtt allows you to read Teleinfo frames from a Serial port and publish them to an Mqtt broker.

teleinfo2mqtt supports both HISTORY and ̀STANDARD modes.

Run the Docker image

The easiest way to start is to deploy the official teleinfo2mqtt image.

docker run -d --name teleinfo2mqtt \
  --device=/dev/ttyUSB0:/dev/ttyUSB0 \
  -e MQTT_URL=mqtt://my_mqtt_broker:1883 \
  -e MQTT_USER="my-super-user" \
  -e MQTT_PASSWORD="my-secret-password" \
  fmartinou/teleinfo2mqtt

Il y a de nombreuses façons de récupérer la téléinfo du compteur Edf, l’un des derniers modules en date est le Lixee mais il a le désavantage de n’être compatible qu’avec certains compteurs. Je vais te montrer comment faire remonter la téléinfo dans home assistant pour 20€ sans modules wifi esp32/8266, avec un micro module usb fabriqué par Charles Hallard le dernier en date est le micro téléinfo v3.0, nommé aussi µtéléinfo.

Pour ceux d’entre-vous qui ont l’habitude de suivre mes articles, vous avez certainement remarqué que les montages à base de téléinformation sont monnaie courante. J’en utilise sur la carte Arduipi ainsi que sur des adaptateurs dédiés tels que MicroTeleinfo ou PiTinfo.

Le montage utilisé jusqu’à présent était le classique à base d’optocoupleur SFH620 de chez Vishai. Il se découple en plusieurs versions telles que SFH620A, ou SFH6206 (chacun en 3 versions notées de 1 à 6 en fonction du courant de fuite entre le collecteur et l’émetteur).

Téléinfo compteur Linky

Ce script python permet de récupérer les informations de votre compteur linky via la liaison série du port usb. Les informations sont ensuite envoyées à Jeedom sur un Virtuel. Ce script a été réalisé pour un raspbery zero mais est facillement adaptable à pour un autre équipement.

Open Shop Channel

Homebrew apps library for the Nintendo Wii.

Linking the world, one Wii at a time

WiiLink is an open source replacement for Nintendo's servers for both WiiConnect24 and Nintendo Wi-Fi Connection, allowing online connectivity for all, even after the original servers were discontinued!