30 Mar 2016, 09:30

Around the Web - March 2016 - Javascript, Citus DB and Dependencies

Javascript

Postgres

Dependencies

27 Jan 2016, 09:30

Around the Web - January 2016 - Website obesity crisis, AngularJS & Postgres

Website Obesity

  • Website obesity crisis : transcript of a talk (video can be seen too from the link) about webperformance, bloated websites and how fat sites become for almost no real value for end users. Long but funny and instructive. It makes you think on how complex, fat and bloated the web is nowadays, in terms of tooling, architecture, code and content.

AngularJS

Postgres

  • The most advanced opensource database of the world, ie Postgres for short, was released as 9.5 version (FR / EN) ; it brings the long expected "UPSERT" features, Row level security and some big-data features (improved index, faster sorts, connection to Hadoop/Cassandra via FDW, etc)
  • In French a deeper view of the 9.5 release (part 1, part 2, part 3) to better understand what contains this release.

 

16 Sep 2015, 09:30

Around the Data - September 2015 - SQL, NoSQL, BigData and streaming

Having new activities around big-data topics from this month, I'll publish here also my findings on this topic.

So the "Around the Web" edition should be still be published on every last wednesday of the month and the "Around the Data" series should be published every wednesday in the middle of the month.

(No)SQL/Big Data

  • In a long interview splitted in two parts, "Where big data is headed and why spark is so big" and "Why NoSQL mattered and SQL still matters", the co-creator of AMPLab (the lab behind Spark at least and other big data tools) review what happend on the last decades with the NoSQL movement, how it enforced traditionnal database to evolve, how it enforced to change all the paradigms around data management and now all the big data evolution. And that SQL still matters :-) A long read but with insights and good points.
  • In the same kind of thoughts, there are some "big data" features in Postgres. Postgres has been used as datamart for a while (but not only) and can be used in some analytics / big data context. So you may start with Postgres first before going further (depend on your context)
  • With "Entretise din't have big data, they have bad data" and "You may not need big data after all", First, it insits on the issue of bad data management both in quantity and accuracy. Then, providing the right data is nice but it's about to provide the right data to the right person to take a decision is better (cf 7-Eleven Japan use case). It's also about clearly defining busines rules but also about more human being skills like coaching around data usage and culture shift / change management to adopt a culture of evidence-based decision making.

Streaming

  • Beyond batch : Streaming 101 : introduction to streaming principles, concepts and methods.
  • NoETL
    • Iin the same way NoSQL movment tends to answer to points that traditionnal database could not face to some extend, there is the same movment regarding ETL (Extract Transform and Load) tools. Instead of ETL, they promote CTP (Consume, Transform, Produce) concept.
    • Current "pitfalls" of ETL are identified as data duplication, possible data loss, costs, complexity and slowliness. Idea is also to remove this intermediary step of the ETL which makes the bridge between two systems.
    • New challenge would be to rely first on strong API to avoid the extract phase and data loss/duplication, then new processing tools to allow close to real-time processing and which will produce outcomes, without requiring this intermediary step represented by the ETL. I requires you swith from a batch logic (processed at a given time) to a flow mechanism.
    • Idea behind NoETL is interesting to review the way you manage and process your data. But it has strong requirements / pre-requisites. It requires your applications, systems and infrastructures being well structured and adapted to such needs.

25 Mar 2015, 09:30

Around the Web - March 2015

Browser

Responsive Web Design (RWD)

HTML5/CSS/Javascript

  • This API is so Fetching : fetch API is to be used for asynchronous actions and is to be more resilient than a XHR (ie ajax) call. Some exemples are given in the blog post ; it can be used from Firefox 39 and Chrome 42 (currently in dev status) but a Fetch Polyfill exists to start using this API from now.
  • CSS Reference which introduces itself as an extensive CSS reference with all the important properties and info to learn CSS from the basics ; this article gives a more introduction on its purpose and how to use it.
  • Meteor, develop faster than a rocket (in French) : an introduction to Meteor  a full stack and isomorphic javascript framework in which you use Javascript both on client and server side. It also uses MongoDB (NoSQL Document Oriented database & schemaless) to store data and it's based on Node.JS. A second article will show how you can create a mobile app easily.

Thoughts

  • Your job is not to write code : Engineers' job is not to write code, Project Managers' job is not to manage project and so on. Our job is to make a better product.
  • A Bug Hero to fight against bug invasion (in French): in an agile team, in each sprint, a developper is commited to do the 1st level support, fix bug and manage incident to avoid disturbing the whole team and sacrifice the sprint. If no bugs, developer is aimed to fix small tasks that are not on the critical path for the sprint dlivery. Interesting both for the disturbing management effect and as it enforces developpers to have a global knowledge of the system, not only his own part.  

SQL

  • Understanding SQL's null : because querying null is not as easy as it may be and also null may not mean null in the way you expect.
  • PoWa (Postgresql Workload Analyser), released as a 2.0 version, provides a better (from what it is said, not tested) monitoring and performance tools on your Postgres 9.4 cluster.

Virtualisation

Compose is a way of defining and running multi-container distributed applications with Docker. Back in December we opened up its design to the community. Based on the feedback from that, Compose will be based on Fig, a tool for running development environments with Docker.

Machine takes you from “zero-to-Docker” with a single command. It lets you easily deploy Docker Engines on your computer, on cloud providers, and in your own data center

Swarm is native clustering for Docker containers. It pools together several Docker Engines into a single, virtual host. Point a Docker client or third party tool (e.g., Compose, Dokku, Shipyard, Jenkins, the Docker client, etc.) at Swarm and it will transparently scale to multiple hosts. A beta version of Swarm is now available, and we’re working on integrations with Amazon Web Services, IBM Bluemix, Joyent, Kubernetes, Mesos, and Microsoft Azure.

  • so now you can orchestrate all your process from zero to production using docker (based) solutions. Even if some products are still in beta so far, a very interesting move !

 

04 Sep 2014, 13:00

Postgresql 9.3.5 pour Debian Wheezy et architecture armhf (raspberry, cubieboard, cubietruck, etc)

La fondation Postgres ne fournit des paquets debian/ubuntu que pour les architectures i386/x86_64 ; Voulant avoir une version 9.3 pour mes projets sur carte Raspberry / Cubietruck, j'ai donc recompilé les paquets sur la base des informations fournies ici que je complète.

### add postgre’s sources to apt
sudo echo "deb-src http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg main" >> /etc/apt/sources.list.d/pgdg.list

### add apt key if not done before
wget –quiet -O – https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -

### Prepare your own environment
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential fakeroot

### Satisfy your dependencies
sudo apt-get build-dep postgresql-9.3
sudo apt-get build-dep postgresql-common
sudo apt-get build-dep postgresql-client-common
sudo apt-get build-dep pgdg-keyring

### !!! ATTENTION -> do not run as root or build will fail on test “can not run tests as root”.
### As it can require quite a long time, you are encourage to have a remote session using screen or tmux for example to avoid network disconnection.
cd /tmp
apt-get source --compile postgresql-9.3
apt-get source --compile postgresql-common
apt-get source --compile postgresql-client-common
apt-get source --compile pgdg-keyring

### create own repository
sudo mkdir /var/local/repository
sudo echo "deb [ trusted=yes ] file:///var/local/repository ./" > /etc/apt/sources.list.d/my_own_repo.list
sudo mv /tmp/*.deb /var/local/repository/
cd /var/local/repository
sudo dpkg-scanpackages ./ > Packages && gzip -f Packages

### Update your package list
sudo apt-get update

### <:o) install postgres
sudo apt-get install postgresql-9.3

### list of generated packages :
libecpg6_9.3.5-1.pgdg70+1_armhf.deb
libecpg-compat3_9.3.5-1.pgdg70+1_armhf.deb
libecpg-dev_9.3.5-1.pgdg70+1_armhf.deb
libpgtypes3_9.3.5-1.pgdg70+1_armhf.deb
libpq5_9.3.5-1.pgdg70+1_armhf.deb
libpq-dev_9.3.5-1.pgdg70+1_armhf.deb
pgdg-keyring_2013.2_all.deb
postgresql_9.3+154.pgdg70+1_all.deb
postgresql-9.3_9.3.5-1.pgdg70+1_armhf.deb
postgresql-9.3-dbg_9.3.5-1.pgdg70+1_armhf.deb
postgresql-client_9.3+154.pgdg70+1_all.deb
postgresql-client-9.3_9.3.5-1.pgdg70+1_armhf.deb
postgresql-client-common_154.pgdg70+1_all.deb
postgresql-common_154.pgdg70+1_all.deb
postgresql-contrib_9.3+154.pgdg70+1_all.deb
postgresql-contrib-9.3_9.3.5-1.pgdg70+1_armhf.deb
postgresql-doc_9.3+154.pgdg70+1_all.deb
postgresql-doc-9.3_9.3.5-1.pgdg70+1_all.deb
postgresql-plperl-9.3_9.3.5-1.pgdg70+1_armhf.deb
postgresql-plpython3-9.3_9.3.5-1.pgdg70+1_armhf.deb
postgresql-plpython-9.3_9.3.5-1.pgdg70+1_armhf.deb
postgresql-pltcl-9.3_9.3.5-1.pgdg70+1_armhf.deb
postgresql-server-dev-9.3_9.3.5-1.pgdg70+1_armhf.deb
postgresql-server-dev-all_154.pgdg70+1_all.deb

have fun.

Si quelqu'un est intérressé par la mise en place d'un dépôt, qu'il se manifeste.

19 Mar 2014, 09:30

Misc - 19/3

UI

  • Brick is a bundle of reusable UI components based on HTML5 and made by Mozilla ; you can read an introduction about it. Seems minimalist to some extend (limited in numbers even if growing) but also very powerful at a qualitative point of view.

DevOps

  • If you are interested in Chef or more widely about IT automation, you can read 3 articles in French on how to run Chef from scratch and cook your first recipes.

Postgres

  • PostgreSQL 9.3.3, 9.2.7, 9.1.12, 9.0.16 and 8.4.20 released! It contains fixes for multiple security issues, as well as several fixes for replication and data integrity issues. It also contains many improvements.
  • Barman seems a very powerful backup and recovery manager for Postgres. Develop by the 2nd Quadrant team which are in the main commiters on Postgres code.
  • Postgresql Studio : a web app to manage your postgres database in a Java container (tomcat, etc). Seems a modern alternative to phpPgAdmin.
  • Postgresql Exercies : to discover Postgres world and train yourself to build SQL queries.

Development

  • Some best practices (in French) about POST vs PUT to use them correctly in your API.
  • The AngularJS team wrote a blog post about what would be in AngularJS 2.0 ; seems very promising both on the improvements part but also what they aim to ease. It's also interesting to see that they inspire from ES6 which is the next version of Javascript (not yet implemented in your browser)
  • Opquast is reviewing their best practices around mobile development til end of March, with a public review. Then final recommendation will be released. If you don't know Opquast (for Open Quality Standards), they released a lot of checklists to asses/review your site.
  • The truth about multiple h1 tags in the HTML5 era : with the change of structure allowed with html5, it's now safe to implement multiple h1 tags.
  • Wonder when/why you should use Node.js for your next project : read this article which explains the paradigm behind Node.js and where it fits best for projects.

18 Sep 2013, 09:30

MySQL/Postgres Roundup 18/9

On MySQL side, or should I say MariaDB one :

  • Google swaps out MySQL, moves to MariaDB : beyond the significative reference for MariaDB and the fact that Google will sustain his effort to patch MariaDB as they did for MySQL, the question that raised for me was : how long will Percona go with MySQL ? Percona's server was seen as an advanced version of MySQL with the inclusion of some patches (like Google's one for performance, etc) and for the tools they provide with/aside (like Percona toolkit or Xtrabackup). It does not seem they plan to make the switch so far...
  • Scaling your database via InnoDB table compression : where you can eliminate slow queries via innodb table compression. Constraints and limits are explained in the post.

On Postgres side :

More generally and even if it's a postgres example which is used, you should use UUIDs for your keys instead of traditionnal keys. Beyond unicity, if you are to use distributed systems, it would be one (or the only ?) way to avoid conflicts.

10 Apr 2013, 22:30

SQL/Database Roundup - 10/4

Spending some time on MySQL these days, time for a SQL/Database roundup :

MySQL

Production

  • If you are interested in MySQL and want to know more about administration, replication, optimisation, etc ; the must read book is High Performance MySQL (I highly recommend it) and you can follow the related blog.
  • Percona provides a bunch of scripts to get information about the status of your database, its health, replication status, etc.
  • Percona provides a bunch of templates/scripts/reports/graphs to integrate with your monitoring solution (Nagios/Cacti) ; it covers mainly MySQL but you also have some for Apache, Nginx, MongoDB, etc.
  • More about Percona MySQL graphs in Cacti which capture all the activity of your MySQL server
  • If you are more interested on Postgres monitoring :

And as a side note, yes, Percona provides also their own MySQL based server with some optimisations and additional features and some other tools

Performance :

  • Indexes : the neglected performance all-rounder : where you will learn obvious things :
    • Defining indexes is part of developper jobs and not DBA
    • Indexes are not taught and poorly documented as it's not strictly part of SQL language
  • Need for speed (in French) : a mix of best practices / tools / methodology on optimisation for your database (mainly postgres)
  • MySQL Tuner : a script that would review your MySQL configuration and makes suggestion to improve performance and stability. Full support up to MySQL 5.1 ; Alike, you also have MySQL Tuning Primer.

05 Apr 2013, 22:27

Postgres > Major security issue > PostgreSQL 9.2.4, 9.1.9, 9.0.13 and 8.4.17 released

For those using Postgres, a minor release was made yesterday fixing some major security issues ; everyone is strongly recommended to upgrade its postgres server immediatly ; especially if your server is accessible from outside of your network ; if you are hosted internally, it's safe to upgrade but not as urgent.

Quoting the announce :

A major security issue fixed in this release, CVE-2013-1899, makes it possible for a connection request containing a database name that begins with "-" to be crafted that can damage or destroy files within a server's data directory. Anyone with access to the port the PostgreSQL server listens on can initiate this request. This issue was discovered by Mitsumasa Kondo and Kyotaro Horiguchi of NTT Open Source Software Center.

Two lesser security fixes are also included in this release: CVE-2013-1900, wherein random numbers generated by contrib/pgcrypto functions may be easy for another database user to guess, and CVE-2013-1901, which mistakenly allows an unprivileged user to run commands that could interfere with in-progress backups. Finally, this release fixes two security issues with the graphical installers for Linux and Mac OS X: insecure passing of superuser passwords to a script, CVE-2013-1903 and the use of predictable filenames in /tmp CVE-2013-1902. Marko Kreen, Noah Misch and Stefan Kaltenbrunner reported these issues, respectively.

I will also remember that the 8.3 version of Postgres is no longer maintained nor supported from end February 2013 and current 8.4 is maintained/supported up to July 2014. More information on the Versioning policy page.

Source :

13 Mar 2013, 22:12

Postgres Roundup - 13/3

While skiing this week, you'll only have one blog post from me this week. So, I let you discover the Postgres world with some interesting resources.

For those who don't know Postgres, it's an opensource object relational database management system, initiated in 1995 and which know a rise since Ingres increased its license feeds and also Oracle bought Sun and thus MySQL. MySQL was a very popular database for Web projects at least (it's the M in the LAMP acronym : Linux Apache MySQL PHP). For a detailed history, cf Postgresql on Wikipedia.

So beyond the official documentation :