Many craftWowSuch createhttps://maciej.lasyk.info/2023-10-01T00:00:00+02:00Iteration 2 of homelab Kubernetes2023-10-01T00:00:00+02:002023-10-01T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2023-10-01:2023/Oct/01/iteration-2-of-homelab-kubernetes/<p><center><img alt="Kubespray" src="https://maciej.lasyk.info/images/homelab-kubernetes-logo1.png" width="350px" ></center></p>
<h1>About this blog series</h1>
<p>A few years passed, since I've initially setup my homelab. I've learnt a lot during this journey - starting from bare-metal planning, network topology (which is completely different, than a cloud-one or even, an office one), through storage solutions, emergency power supplies, uplink redundancy, Kubernetes management and performance tuning, various cluster services and finishing on writing own Kubernetes controllers.</p>
<p>So, recently I decided, that I will upgrade this whole cluster, from an old 1.21.x to some recent one, 1.27. However, upgrading k8s is not an easy task, especially, when there're many services already running there. I decided, that the quickest, most reliable and least risky way would be to simply, create a new cluster, and simply, migrate all required services there.</p>
<p>So this all got me thinking, that it's a great opportunity to actually describe this story in a few blog posts. This would be a good starting point, for preparing some meetup presentation in the future. I know - there're plenty of homelabs already there. I believe however, that each story is worth telling, as there's always something to learn from. Especially, when the story describes bunch of experiences.</p>
<h1></h1>Notes on Joplin and Evernote2023-02-25T00:00:00+01:002023-02-25T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2023-02-25:2023/Feb/25/notes-on-joplin-and-evernote/<p><center><img alt="Joplin" src="https://maciej.lasyk.info/images/joplin-logo.png" width="150px" ></center></p>
<h1>Joplin and Evernote? Wait, what?</h1>
<p>Both tools are great for storing information. I'd been using <a href="https://evernote.com/">Evernote</a> since 2012, and it's been my central database of knowledge. It provides very useful features, which make fetching, saving, storing and accessing information effortless. It is a SAAS solution, subscription-based and closed source. Some of my favorite features:</p>
<ul>
<li>With Evernote WebClipper (browser plugin) you can save a whole webpage (or a simplified version, stripped from rubbish, menus, ADs etc). So, no more need to search for <em>that</em> webpage anymore.</li>
<li>This webclipper also adds a widget to search results (Google, DDG and others). This widget presents notes, which fit your search query from the search engine. Cool, isn't it?</li>
<li>Taking notes during various conferences, meetups, workshops is much easier, as you may simply take photos of slides. Evernote has OCR, so basically, you may search for words in images</li>
<li>Evernote has mobile apps as well as desktop ones (plus web editor)</li>
<li>Thousands of other features that... I haven't really used, as didn't feel like it would help my productivity.</li>
</ul>
<p><a href="https://joplinapp.org/">Joplin</a> is a very similar app, but it's an open-source project. And you may host it yourself.</p>
<h1>Why migrate?</h1>
<p>For the last few years, Evernote apps performance has been basically getting worse. IOS app opens in a few seconds (instead of like 0.4s or smt), then starts loading data (wait, what?) and 30s later I can finally start searching for something. Moreover, my subscription plan didn't include features like.. keyboard shortcuts customization. Yes, after one of the recent updates in the billing plans, you may only change keyboard shortcuts on some specific plan. I could live with that, but go figure, in my native language, letters like ęąć are typed via pressing right alt and the corresponding letter. And you know what? Evernote overwrote somehow those shortcuts on the OS level. So, each time I tried to type one of those letters (in any application) Evernote-related feature/app opened. And I wasn't able to change that - I needed to shut down the app completely. Really, what a mess.</p>
<p>And also, I've always been lacking some basic features:</p>
<ol>
<li>There was no Markdown native support (Joplin has it)</li>
<li>There was no simple way to actually turn a note into a blog entry on any of my blogs. I always wanted to work on blog-posts on Evernote, and somehow, simply publish it to the blog. There is the Evernote API, which I might have used for that, but again - I would need to convert notes to Markdown.</li>
<li>I really need to easily backup all my notes somehow, and be able to restore it - best in an automated way. Evernote has export features - that are a feature of the desktop app. No easy way for automation.</li>
<li>Also, with Evernote I don't really feel that I own the data. I always felt that whatever is stored in Evernote, is a subject of turning public someday. So saving private stuff there is not a thing, that I feel comfortable with. Not that I haven't done that.</li>
</ol>
<p>So yeah, after many years, I finally decided to perform the migration.</p>
<h1>Hosting Joplin on Kubernetes</h1>
<p>This is actually an easy thing. As there is no an official Helm chart, I decided to create my own (which I might actually publish at some stage; ping me if you'd like me to). What I did is basically wrote a simple deployment wrapping <a href="https://hub.docker.com/r/joplin/server">this official Joplin container image</a>. A few assumptions here:</p>
<ol>
<li>I use an external Postgres database. Joplin saves data locally (desktop app) in sqlite database. Then syncs with the Joplin server (cloud or self-hosted). In my case, I decided to go w/Postgres, as from my research, it occurred, this is the most performant way.</li>
<li>Postgres has its downsides. You need to host and maintain it ofc.</li>
<li>I have 3 users working on this Joplin (basically, my family). Joplin is accessible only via our VPN (Mikrotik router -> Wireguard VPN). This is imo the best possible solution to provide security for your data. Downside - all users need to have this VPN on mobile devices and desktops/laptops.</li>
<li>There isn't any additional persistence requirement but the database. No PVCs or anything.</li>
<li>I use only 1 replica atm, as this is enough. Even while importing data, there were no serious performance issues - it just took some time (around 4GB of data to push into Postgres).</li>
</ol>
<p>The actual Migration from Evernote was a manual task. I needed to <a href="https://help.evernote.com/hc/en-us/articles/208313528-Back-up-and-restore-your-data-in-Evernote-for-Windows">export notebooks</a> via Evernote desktop app, and later import it into Joplin desktop app. What was very cool, is that it imported it directly into Markdown format - and did it very good <3</p>
<p>Also, remember, that even though you self-host the Joplin, enabling full encryption might be worth the effort. Basically, Joplin syncs and holds all data locally by default. This means, that e.g. all your unencrypted notes will be saved on your mobile phone - let's say on IOS. So, iCloud backup will save it also (or any other backup solution).</p>
<h1>Notes on Joplin high availability</h1>
<p>Self-hosting Joplin might seem to be a delicate thing. What about a situation, where you're on holidays, or simply out of home, and you need to access the Joplin, but there is a power outage, or an internet disruption at home?</p>
<p>Well, firstly - Joplin notes are synced and can be accessed offline. So in the worst - case scenario, you have your notes, but not updated to the latest versions.</p>
<p>But if you really want to have a truly disaster-recovery plan, consider my approach. I have the backup job, which saves postgres backups to a remote location, and there, it imports it every day (this is also a good way to test backups). And I have the secondary Joplin instance somewhere. If my home stops working, I can switch the proper DNS entry, and that would do the trick. But:</p>
<ol>
<li>No VPN there (for simplicity)</li>
<li>Afterward, I'd need to somehow manage synchronization from the secondary place to my home. It might get very tricky. So.. well. I haven't used this yet (but when I set it up and confirmed the POC), as I haven't had a need for that. But I have this option.</li>
</ol>
<h1>Joplin API</h1>
<p>Another cool stuff is that Joplin app <a href="https://joplinapp.org/api/references/rest_api/">provides data API</a>. So in theory, you might work on some automation here. Exposing the API from the server hosted in k8s POD <a href="https://discourse.joplinapp.org/t/does-joplin-server-have-the-api/19374/7">might be a tricky thing</a> (as normally, it is exposed only via the desktop or cli app), but I think this is doable.</p>
<h1>Joplin FUSE filesystem project</h1>
<p>What I really lack in Joplin (that Evernote also doesn't provide) is the possibility to mount notes as text files on the filesystem. The thing is that I really find it helpful to have some of those notes being searchable from the context of IDE I currently work in. E.g. when working in the GoLand or vscode, I like having my notes added as an additional library to the project. This way I can easily search for information I saved on a specific subject, or code templates etc. Up to now, I have basically used a private git repository for storing that kind of information, and it was separated from Evernote. Just because of this particular use - case.</p>
<p>So now, in Joplin it is not possible to achieve, but I can work on this feature myself. I have started <a href="https://discourse.joplinapp.org/t/mounting-notes-as-files-on-the-filesystem/29543">this thread</a> on the Joplin forum, and basically, I started working on the <a href="https://github.com/docent-net/go-joplinfs-fuse">FUSE virtual filesystem</a>, that will make it work cross-platform.</p>Migrating Mastodon storage to S3-compatible2023-01-08T00:00:00+01:002023-01-08T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2023-01-08:2023/Jan/08/migrating-mastodon-storage-to-s3-compatible/<p><center><img alt="Mastodon" src="https://maciej.lasyk.info/images/mastodon.jpg" width="550px" ></center></p>
<p>If you're just curious, how to perform the migration, navigate to bottom to the <a href="#the-solution-howto">The solution HOWTO</a> section.</p>
<p>This blog entry is written in a way, to be understood by even entry-level admins, who maintain own Mastodon instances. So this time I attached even some screenshots to make things more clear.</p>
<h1>The problem</h1>
<p>Recently, I enabled a few relays on my Mastodon instance to populate our content in the network. However, after that action, storage usage grew rapidly:</p>
<p><center><img alt="Mastodon-storage-problem" src="https://maciej.lasyk.info/images/mastodon-2-2.png" width=550></center></p>
<p>You see, how quick it was? Since then, I have increased the capacity of that disk volume a few times (using QNAP for k8s storage here), but this alert just wouldn't go away:</p>
<p><center><img alt="Mastodon-storage-problem" src="https://maciej.lasyk.info/images/mastodon-2-1.png" width=550></center></p>
<p>I have the <strong>media-remove</strong> process enabled, which deletes cache data older than 3 days now. But still - the grow rate is faster than data removal:</p>
<p><center><img alt="Mastodon-storage-problem" src="https://maciej.lasyk.info/images/mastodon-2-3.png" width=550></center></p>
<h1>Env setup</h1>
<p>My Mastodon instance is a Kubernetes deployment done using <a href="https://github.com/mastodon/chart">an upstream helm chart</a>, which I edited a bit to fit my use case. I have disabled using Postgres subchart and deployed it separately using <a href="https://bitnami.com/stack/postgresql-ha/helm">Bitnami postgres-ha helm chart</a>.</p>
<p>The storage is provided by QNAP hardware storage. I have 2x500GB SSDs there in RAID1 and I expose this storage via <a href="https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner">nfs-subdir-external-provisioner</a>. As I have enabled snapshots for some workloads, in practice I have around 400GB of usable storage. <strong>Mastodon storage was quickly raising, almost up to 100GB</strong>, which is almost 25% of the whole space. Basically, the cost of hosting Mastodon, from this perspective, grew considerably.</p>
<h1>Finding the solution</h1>
<p>I thought about a few solutions here:</p>
<ol>
<li>Shorten the cache TTL by lowering the number of days in the <strong>media-remove</strong> job. I have it set to 3 days, but could lower to e.g., 1 day. However, I realized that several tags I follow are not updated that recently. Thus, on my timeline, it's normal, that I see posts from a few days back. So that 3 days seems really reasonable, and possibly, I'm thinking about even elevating this number to e.g., 5 days. So, this approach is not the best course of action for my situation and requirements.</li>
<li>Resize the volume to e.g. 200GB. This would be far too expensive. This QNAP price (including disks) was around $400. If I sacrifice half of its storage just for Mastodon, the cost would be around $200, plus maintenance, energy etc. Imo, it's way too expensive for this kind of workload (even though it's a one-off cost; it doesn't amortize too well).</li>
<li>Migrate to secondary QNAP with standard HDDs. I have a secondary device, with higher storage (2x2TB). But this is way slower hardware (same QNAP, but slower disks, normal hard-drives). I use it mainly for storing snapshots, backups and some other, heave, rarely accessed data. So, this wouldn't be the best course of action, as it would introduce a much higher pressure on those drives.</li>
<li>Migrate to the cloud. Mastodon provides the ability to write/read assets and other static files to/from S3-compatible drive. I have done a research here, and compared 3 providers: <a href="https://aws.amazon.com/s3/pricing/">AWS S3</a>, <a href="https://developers.cloudflare.com/r2/platform/pricing/">Cloudflare R2</a> and <a href="https://www.idrive.com/e2/pricing">iDrive E2</a>. The last is the winner here - it's the cheapest one, it provides some basic ACLs (public read-only for my use-case) <strong>and there are no costs related to data transfers</strong> (in and out so ingress and egress).</li>
</ol>
<p><center><img alt="Mastodon-storage-problem" src="https://maciej.lasyk.info/images/mastodon-2-4.png" width=550></center></p>
<h1>Exploring iDrive E2</h1>
<p>I have found this service just recently, while searching for a possible storage vaults for my QNAP backups/snapshots. I have run a proof-of-concept using it:</p>
<ol>
<li>Created a bucket for Mastodon</li>
<li>Enabled public-read-only permissions (it's possible to upload a custom IAM policy also as it's compatible with S3)</li>
<li>Integrated it with Cloudflare CDN (by generating Cloudflare TLS certs and integrating DNS subdomains)</li>
<li>Copied a small set of Mastodon data into this new bucket using <strong>aws s3 sync</strong></li>
<li>Switched Mastodon to read/write into this S3 to confirmed it works properly. <strong>It works properly!</strong></li>
</ol>
<blockquote>
<p>So, my estimated costs now (assuming 200GB of storage use) is around.. <strong>200 * $0.004 = $0.8</strong> monthly (minus the first 10GB which is free). So it's less than a dollar per month, <strong>making it $9.6 YEARLY for those 200GB</strong>. Way, way cheaper, than hosting it on my SSDs.</p>
</blockquote>
<p><a name="the-solution-howto"></a></p>
<h1>The solution HOWTO</h1>
<p>The process is rather straightforward, just customize to match your environment.</p>
<ol>
<li>Register an <a href="https://www.idrive.com/e2/">iDrive E2 account</a> and enable billing by upgrading your plan (whether yearly or monthly). Without a billed account, you won't be able to open the bucket to the world (make it semi-public). Choose a region close to your site.</li>
<li>When it's done, create a new bucket. As you can see, there is an option to enable Public access with read-only setting. IMO, this is not too secure, as it may also provide a user to iterate over files (fetch directory index) and that is not required (and can be dangerous; there is no need for anyone to fetch index of your storage). Thus, its much safer to specify a strict IAM policy, that allows to only fetch a particular file knowing it's URL (and Mastodon knows it): <center><img alt="Mastodon-storage-problem" src="https://maciej.lasyk.info/images/mastodon-2-5.png" width=550></center></li>
</ol>
<div class="highlight"><pre><span></span><span class="o">{</span>
<span class="w"> </span><span class="s2">"Version"</span>:<span class="w"> </span><span class="s2">"2012-10-17"</span>,
<span class="w"> </span><span class="s2">"Statement"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="s2">"Sid"</span>:<span class="w"> </span><span class="s2">"AllowPublicRead"</span>,
<span class="w"> </span><span class="s2">"Effect"</span>:<span class="w"> </span><span class="s2">"Allow"</span>,
<span class="w"> </span><span class="s2">"Principal"</span>:<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="s2">"AWS"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="s2">"*"</span>
<span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="o">}</span>,
<span class="w"> </span><span class="s2">"Action"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="s2">"s3:GetObject"</span>
<span class="w"> </span><span class="o">]</span>,
<span class="w"> </span><span class="s2">"Resource"</span>:<span class="w"> </span><span class="o">[</span>
<span class="w"> </span><span class="s2">"arn:aws:s3:::bucket-name/*"</span>
<span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="o">}</span>
<span class="w"> </span><span class="o">]</span>
<span class="o">}</span>
</pre></div>
<ol>
<li>Generate Access Keys for your account (side menu, <strong>Access Keys</strong>): <center><img alt="Mastodon-storage-problem" src="https://maciej.lasyk.info/images/mastodon-2-6.png" width=550></center></li>
<li>Now, the easiest way to use those keys is to export it to env vars, e.g.:</li>
</ol>
<div class="highlight"><pre><span></span><span class="nb">export</span><span class="w"> </span><span class="nv">AWS_ACCESS_KEY_ID</span><span class="o">=</span><span class="s2">"<your access key>"</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">AWS_SECRET_ACCESS_KEY</span><span class="o">=</span><span class="s2">"<your secret access key>"</span>
</pre></div>
<ol>
<li>Verify it works properly. <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">Install aws-cli</a> on your terminal</li>
<li>Having aws-cli installed and security credentials exported to environment variables, you may verify it all works properly. Firt take note of your region endpoint (you can see it in the E2 drive dashboard, assigned to the region you chose): <center><img alt="Mastodon-storage-problem" src="https://maciej.lasyk.info/images/mastodon-2-7.png" width=550></center></li>
<li>Now, you should be able to run the following command. It lists files in your bucket (the list will be probably empty, as you didn't upload there yet anything): <code>aws s3 --endpoint-url https://<your-endpoint-url> ls s3://<your-bucket-name>/</code></li>
<li>Try sending some image file: <code>aws s3 --endpoint-url https://<your-endpoint-url> cp image.jpg s3://<your-bucket-name>/image.jpg</code></li>
<li>Now, try accessing this image <strong>using private mode of your browser</strong>. This way, you will be unauthorized from E2 drive perspective, but as you just made this drive access to specific files allowed publically, it should just work. The url for your file can be built in two ways:<ol>
<li>You may use region endpoint url (the one mentioned earlier). This way, the URL looks like <strong>https://<your-region-endpoint>/<your-bucket-name>/image.jpg</strong></li>
<li>You may also use the bucket endpoint, which looks like <strong>https://<your-bucket-name>.<your-region-endpoint>/image.jpg</strong></li>
<li>Both above methods should work just fine at this point.</li>
</ol>
</li>
<li>Now, if above works fine, the time-consuming part. You'll need to copy your existing data to S3. As in my case, Mastodon is a k8s deployment, I needed to create a pod (that includes aws-cli binary) and attach the Mastodon storage disk. The pod definition was as follows:</li>
</ol>
<div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v1</span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Pod</span>
<span class="nt">metadata</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">aws-cli-copy</span>
<span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mastodon</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">system-pv</span>
<span class="w"> </span><span class="nt">persistentVolumeClaim</span><span class="p">:</span>
<span class="w"> </span><span class="nt">claimName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mastodon-system</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">assets-pv</span>
<span class="w"> </span><span class="nt">persistentVolumeClaim</span><span class="p">:</span>
<span class="w"> </span><span class="nt">claimName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mastodon-assets</span>
<span class="w"> </span><span class="nt">containers</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">datacopy</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">amazon/aws-cli</span>
<span class="w"> </span><span class="nt">env</span><span class="p">:</span>
<span class="w"> </span><span class="c1"># generally it's safer to read secret values from k8s secrets, but for</span>
<span class="w"> </span><span class="c1"># one-shot job, and under my infra circumstances, this is fine:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">AWS_ACCESS_KEY_ID</span>
<span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="s">"<your-access-key-id>"</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">AWS_SECRET_ACCESS_KEY</span>
<span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="s">"<your-secret-access-key>"</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"sleep"</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"36000"</span>
<span class="w"> </span><span class="nt">volumeMounts</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="s">"/mnt/mastodon-system"</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">system-pv</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="s">"/mnt/mastodon-assets"</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">assets-pv</span>
</pre></div>
<ol>
<li>Now, after pod was created (<code>kubectl apply -n mastodon -f pod.yaml</code>) I just entered its terminal (<code>k exec -it -n mastodon aws-cli-copy -- bash</code>)</li>
<li>Now, the data-sync command. From the directory, where all mastodon-system files were stored (in my case <strong>/mnt/mastodon-system</strong>) I just run a command: <code>aws s3 --endpoint-url https://<your-endpoint-url> sync . s3://netrunner-masto/ --dryrun</code> - mind the <strong>dryrun</strong> part. It tells the command to just display, what it would change, instead of sending files. It's better to verify it's doing what we want it to do, before running the actual command. After verification, just remove the <strong>dryrun</strong> part and run the command again.</li>
<li>After you synchronized those files, now you need to reconfigure Mastodon properly, to make it use this S3-compatible storage. As I use the <a href="https://github.com/mastodon/chart">helm chart</a>, the proper values to change were:</li>
</ol>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="nt">s3</span><span class="p">:</span>
<span class="w"> </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">access_key</span><span class="p">:</span><span class="w"> </span><span class="s">"<your-access-key>"</span>
<span class="w"> </span><span class="nt">access_secret</span><span class="p">:</span><span class="w"> </span><span class="s">"<your-secret-access-key>"</span>
<span class="w"> </span><span class="nt">existingSecret</span><span class="p">:</span><span class="w"> </span><span class="s">""</span>
<span class="w"> </span><span class="nt">bucket</span><span class="p">:</span><span class="w"> </span><span class="s">"<your-bucket-name>"</span>
<span class="w"> </span><span class="nt">endpoint</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://<your-region-endpoint-url></span>
<span class="w"> </span><span class="nt">hostname</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain"><your-region-endpoint-url></span>
<span class="w"> </span><span class="nt">region</span><span class="p">:</span><span class="w"> </span><span class="s">""</span>
<span class="w"> </span><span class="nt">alias_host</span><span class="p">:</span><span class="w"> </span><span class="s">"<your-region-endpoint-url>/<your-region-bucket>"</span>
</pre></div>
<ol>
<li>It's actually better&safer to put access key/secret into a secret and use <strong>existingSecret</strong> in the above yaml config.</li>
<li>If you're not using k8s, but standard VM/bare-metal deployment, <a href="https://docs.joinmastodon.org/admin/config/#s3">this documentation</a> explains which env vars to change in order to enable S3. I believe it would look like:</li>
</ol>
<div class="highlight"><pre><span></span><span class="nv">S3_ENABLED</span><span class="o">=</span><span class="nb">true</span>
<span class="nv">S3_BUCKET</span><span class="o">=</span><your-bucket-name>
<span class="nv">S3_PROTOCOL</span><span class="o">=</span>https
<span class="nv">S3_HOSTNAME</span><span class="o">=</span><your-region-endpoint-url>
<span class="nv">S3_ENDPOINT</span><span class="o">=</span>https://<your-region-endpoint-url>
<span class="nv">S3_ALIAS_HOST</span><span class="o">=</span><your-region-endpoint-url>/<your-region-bucket>
<span class="nv">AWS_ACCESS_KEY_ID</span><span class="o">=</span><your-access-key>
<span class="nv">AWS_SECRET_ACCESS_KEY</span><span class="o">=</span><your-secret-access-key>
</pre></div>
<ol>
<li>Now, after restarting/redeploying Mastodon, it all should just work.</li>
<li>One more thing left here. I had this mastodon-assets volume, which I didn't migrate. But assets stored there are still hosted properly, so I believe those are properly provided by the webserver. The thing is that it may be ineffective to host static files via this webserver. However, as I use CDN for caching static content - I don't really care. If this start to be problematic from the performance side, I'll take a closer look into it.</li>
</ol>
<h1>CDN considerations</h1>
<p>The above setup has one disadvantage. Having the following setup:</p>
<blockquote>
<p>alias_host: <your-region-endpoint-url>/<your-region-bucket></p>
</blockquote>
<p>means, that each time a content is downloaded, it is done directly from the S3-compatible storage. In case of eDrive E2 it doesn't really matter, as there are no fees for egress communication (downloading files from your drive). But if this were AWS S3, it would mean that you pay for that outgoing traffic.</p>
<p>To prevent this, we can use a Content Delivery Network (CDN) which acts as a caching layer in front of this S3-compatible drive. It adds reliability and speeds up content downloading (CDNs have many nodes across the globe, and usually the client is connected to the closest one, meaning, lower roundtrip times).</p>
<p>But this will be a subject of another blog post.</p>Unmounting QNAP system volume to perform e2fsck2023-01-05T00:00:00+01:002023-01-05T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2023-01-05:2023/Jan/05/unmounting-qnap-system-volume-to-perform-e2fsck/<p><center><img alt="QNAP TS-231K" src="https://maciej.lasyk.info/images/qnap-ts-231k.jpg"></center></p>
<h1>The problem</h1>
<p>This is just a quick note related to a problem I'm having from time to time, during maintenance of my QNAP volumes (QNAP TS-231K).</p>
<p>Sometimes I cannot resize my volumes as <strong>Filesystem is not clean. Please perform a check disk operation</strong>. However running the <strong>Check filesystem</strong> operation fails: <strong>Examination failed (Cannot unmount disk)</strong></p>
<h1>The solution</h1>
<p>If it's safe, you may try rebooting the device and try again running the <strong>Check filesystem</strong> operation. This sometimes is enough.</p>
<p>However, often it just doesn't help. Usually this is the case with the volume marked by QNAP as <strong>the system volume</strong>. Basically, when creating a first volume on the storage pool, QNAP marks it as the system volume, and puts there some data.</p>
<p>In that case, my goto procedure is:</p>
<ol>
<li>Login via ssh to the user account <strong>and enter / directory</strong>: <code>cd /</code> (leave your homedir)</li>
<li>Sudo to the admin account <strong>and enter / directory</strong>: <code>cd /</code> (leave your homedir)</li>
<li>Stop all services #1: <code>/etc/init.d/services.sh stop</code></li>
<li>Stop all services #2: <code>/etc/init.d/opentftp.sh stop</code></li>
<li>Stop all services #3: <code>/etc/init.d/Qthttpd.sh stop</code></li>
<li>Find the device and it's mount point, you're having problems with. This command will help you make the choice: <code>mount | grep -i cachedev</code>. The device handle should look like <strong>/dev/mapper/cachedev1</strong> and the mount point like <strong>/share/CACHEDEV1_DATA</strong> (ofc choose a proper one fitting your case, it might be called differently)</li>
<li>Verify whethere there are still some opened files on this device: <code>lsof | grep <mount-point></code>, e.g.: <code>lsof | grep /share/CACHEDEV1_DATA</code></li>
<li>If there are still opened files, kill processes that hold those file-handles. The second column of the lsof command shows the pid (process ID), so the command would look like: <code>kill 1234</code> where 1234 is the PID. Repeat this step until lsof command shows nothing (no more opened files).</li>
<li>Remember, that I wrote to leave your homedir after logging via SSH (both for normal user and admin?). This is very important step - if you enter a homedir, which is mounted inside of this volume, you may not see this thing as an opened file in the lsof output, but still, you won't be able to unmount the device properly!</li>
<li>Now you may try unmounting the device: <code>umount <device-handle></code>, e.g.: <code>umount /dev/mapper/cachedev1</code></li>
<li>If this fails, you may try enforcing it: <code>umount -l <device-handle></code></li>
<li>If umount was successfull, you may try running check command: <code>e2fsck -f -v -C 0 <device-handle></code>, e.g.: <code>e2fsck -f -v -C 0 /dev/mapper/cachedev1</code></li>
</ol>
<p>That usually works for me.</p>Mastodon performance tuning / postgres + pgpool2022-12-29T00:00:00+01:002022-12-29T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2022-12-29:2022/Dec/29/mastodon-performance-tuning-postgres-pgpool/<p><center><img alt="Mastodon" src="https://maciej.lasyk.info/images/mastodon.jpg" width="550px" ></center></p>
<h1>The problem</h1>
<p>Recently, my Mastodon instance started flapping - it was happening more and more often. I started debugging the thing, and found out, that there are problems with <a href="https://pgpool.net">pgpool</a> - it was timing out when running heartbeats:</p>
<div class="highlight"><pre><span></span>Liveness<span class="w"> </span>probe<span class="w"> </span>failed:<span class="w"> </span>pgpool<span class="w"> </span><span class="nv">INFO</span><span class="w"> </span><span class="o">==</span>><span class="w"> </span>Checking<span class="w"> </span>pgpool<span class="w"> </span>health...
psql:<span class="w"> </span>timeout<span class="w"> </span>expired
</pre></div>
<p>As it is a Kubernetes setup, and this heartbeat is a livenessProbe, when it fails - the pod is restarted, causing all applications to fail when connecting to the database. BRUH.</p>
<h1>Env setup</h1>
<p>My Mastodon instance is a Kubernetes deployment done using <a href="https://github.com/mastodon/chart">upstream helm chart</a> which I edited a bit to fit my usecase. I have disabled using Postgres subchart and deployed it separately using <a href="https://bitnami.com/stack/postgresql-ha/helm">Bitnami postgres-ha helm chart</a>.</p>
<p>I've git 3 postgres instances in clustered mode (using <a href="https://repmgr.org/">repmgr</a> for managing replication). Loadbalancing and connection pooling is done via <a href="https://pgpool.net">pgpool</a>.</p>
<h1>Digging into the problem</h1>
<p>The liveness probe is a k8s special kind of check, which tells whether the service is working fine. In this particular case, it is configured to run a bash script <a href="https://github.com/bitnami/containers/blob/main/bitnami/pgpool/4/debian-11/rootfs/opt/bitnami/scripts/pgpool/healthcheck.sh">/opt/bitnami/scripts/pgpool/healthcheck.sh</a>. This script was timing out, so I tried to run it manually to confirm, there is a problem. It was randomly timing out when connecting to the postgres via pgpool (which stands in front of the database backends).</p>
<p>I had a quick suspicion, that the root cause might be the max_connection reached in Postgres backend servers. I verified this, but servers were performing good at that time:</p>
<div class="highlight"><pre><span></span><span class="n">postgres</span><span class="o">=#</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="n">max_conn</span><span class="p">,</span><span class="n">used</span><span class="p">,</span><span class="n">res_for_super</span><span class="p">,</span><span class="n">max_conn</span><span class="o">-</span><span class="n">used</span><span class="o">-</span><span class="n">res_for_super</span><span class="w"> </span><span class="n">res_for_normal</span><span class="w"> </span><span class="k">from</span>
<span class="p">(</span><span class="k">select</span><span class="w"> </span><span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="n">used</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">pg_stat_activity</span><span class="p">)</span><span class="w"> </span><span class="n">t1</span><span class="p">,</span>
<span class="p">(</span><span class="k">select</span><span class="w"> </span><span class="n">setting</span><span class="p">::</span><span class="nb">int</span><span class="w"> </span><span class="n">res_for_super</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">pg_settings</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">name</span><span class="o">=</span><span class="err">$$</span><span class="n">superuser_reserved_connections$$</span><span class="p">)</span><span class="w"> </span><span class="n">t2</span><span class="p">,</span>
<span class="p">(</span><span class="k">select</span><span class="w"> </span><span class="n">setting</span><span class="p">::</span><span class="nb">int</span><span class="w"> </span><span class="n">max_conn</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">pg_settings</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">name</span><span class="o">=</span><span class="err">$$</span><span class="n">max_connections$$</span><span class="p">)</span><span class="w"> </span><span class="n">t3</span><span class="p">;</span>
<span class="w"> </span><span class="n">max_conn</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">used</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">res_for_super</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">res_for_normal</span>
<span class="c1">----------+------+---------------+----------------</span>
<span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">40</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">57</span>
<span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="k">row</span><span class="p">)</span>
</pre></div>
<p>It was similar on rest of Postgres nodes, so "no luck" this time. It was something different, than an easy problem of max connections reached.</p>
<p>However this made me thinking, that there is this <strong>pgpool</strong> acting as a loadbalancer and connection pooler, standing in front of all postgres instances. As I deployed it with default values, I decided to take a look into details here.</p>
<p>I found this <a href="https://github.com/bitnami/charts/issues/6094">Github issue</a> which was basically describing my problem. The main fix describe was:</p>
<blockquote>
<p><strong>Defined initial child process and the connection lifetime</strong>.</p>
</blockquote>
<p>Actually, the mostsuggested fix was to increase the default <strong>num_init_children</strong> value (32) to something more:</p>
<blockquote>
<p>numInitChildren: 64</p>
</blockquote>
<h1>How to get a proper value for num_init_children?</h1>
<p>So actually, there is this great document on this subject <a href="https://www.pgpool.net/mediawiki/index.php/Relationship_between_max_pool,_num_init_children,_and_max_connections0">Relationship_between_max_pool,_num_init_children,_and_max_connections</a>.</p>
<p>The formula to understand here is:</p>
<blockquote>
<p><strong>max_pool*num_init_children <= (max_connections - superuser_reserved_connections)</strong></p>
</blockquote>
<p>And now, to understand this correctly:</p>
<blockquote>
<p><strong>max_pool</strong> parameter configures how many connections to cache <em>per child</em>. So if num_init_children is configured to 100, and max_pool is configured to 3, then pgpool can potentially open 300 (=3*100) connections to the backend database.</p>
</blockquote>
<p>Especially this one:</p>
<blockquote>
<p>A child process opens a new backend connection only if the requested [user,database] pair is not already in the cache. So if the application uses only one user to connect to only one database, say [pguser1,pgdb1], then each child will continue to reuse the first connection and will never open a second connection, so in effect pgpool will open no more than 100 backend connections even though max_pool is set to 3.</p>
</blockquote>
<p>So just to summarize: <strong>num_init_children</strong> can be set to a number of <strong>max_connections - superuser_reserved_connections</strong> (no matter how many postgres workers you have). And we can set safely <strong>max_pool</strong> to <strong>1</strong> <strong>because Mastodon operates only on one database</strong>. So according to paragraphs mentioned above, the formula turns into:</p>
<blockquote>
<p><strong>num_init_children <= (max_connections - superuser_reserved_connections)</strong></p>
</blockquote>
<p>So e.g. in my case I set:</p>
<div class="highlight"><pre><span></span><span class="c1"># pgpool</span>
<span class="nv">max_pool</span><span class="o">=</span><span class="m">1</span>
<span class="nv">num_init_children</span><span class="o">=</span><span class="m">90</span>
<span class="c1"># postgres</span>
<span class="nv">max_connections</span><span class="o">=</span><span class="m">100</span>
<span class="nv">superuser_reserved_connections</span><span class="o">=</span><span class="m">3</span>
</pre></div>
<p>And this still gives me around 7 connections to be used by any other applications (e.g. monitoring etc).</p>
<h1>Another performance considerations</h1>
<ul>
<li>Tuning livenessprobe: imo increasing timeouts on livenessProbes is not a good solution - it just tries to omit the underlying problem, which will be still there.</li>
<li>When deploying on k8s, set <strong>reserved_connections</strong> to 1 (default: 0). More on this <a href="https://github.com/bitnami/charts/pull/8398">here</a></li>
<li>pgpool heartbeat script timeouts might be also about low entropy level on the pgpool container; <a href="https://github.com/bitnami/bitnami-docker-pgpool/issues/35#issuecomment-1039935634">see this for details</a> - this was not my case.</li>
<li>Running performance test is always a good idea (to confirm pgpool / max_connections settings): <code>pgbench -c 64 -j 4 -t 1000 -f queries.sql</code></li>
<li>Verify LB distribution:</li>
<li><code>pgbench -U <username> -c 10 -T 10 -S</code></li>
<li><code>psql -U <username> -W -c "show pool_nodes"</code></li>
<li>Look at <code>select_cnt</code> of <code>show pool_nodes</code> and draw your conclusions</li>
</ul>
<h1>Interesting URLs on this subject</h1>
<ul>
<li><a href="https://www.ashnik.com/scaling-up-and-load-balancing-your-postgresql-cluster-using-pgpool-ii/">https://www.ashnik.com/scaling-up-and-load-balancing-your-postgresql-cluster-using-pgpool-ii/</a></li>
<li><a href="https://www.pgpool.net/mediawiki/index.php/Relationship_between_max_pool,_num_init_children,_and_max_connections">Relationship_between_max_pool,_num_init_children,_and_max_connections</a></li>
<li><a href="https://hackernoon.com/postgresql-connection-pooling-and-load-balancing-with-pgpool-ii-cluster-edbbb7959ef1">https://hackernoon.com/postgresql-connection-pooling-and-load-balancing-with-pgpool-ii-cluster-edbbb7959ef1</a></li>
</ul>
<h1>Tuning Mastodon / performance notes</h1>
<p>This is one of a series of notes I created during one of performance tuning
sessions done on my own <a href="https://n3trunn3r.org/">Mastodon instance</a>.</p>
<p>I'm planning to write a broad article about this subject, but for now, just a tiny note about on of the things I needed to work on.</p>How to connect to Extreme Networks Summit X450a-48t2022-05-02T00:00:00+02:002022-05-02T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2022-05-02:2022/May/02/how-to-connect-to-extreme-networks-summit-x450a-48t/<p><center><img alt="Extreme Networks Summit X450a-48t" src="https://maciej.lasyk.info/images/x450a-48t.png" width="550px" ></center></p>
<h1>What is it about?</h1>
<p>Some time ago I was lucky to buy this <strong>Extreme Networks Summit X450a-48t</strong> at
a very good price (less than a $100). I needed to extend capacity of my
internal network way beyond previous setup.</p>
<p>So a friend of mine suggested me this wirespeed hardware, that can also provide
L3 support for BGP. It is important, as I use MetalLB on my Kubernetes cluster
in BGP mode.</p>
<p>When the switch arrived, it was clear, that I will need to connect to the
management console of this switch, as I needed to replace firmware with another
one, and reset all configurations to defaults.</p>
<h1>Connecting to the switch</h1>
<p>Three ways of connecting to management console:</p>
<ol>
<li>Using serial/RS-232 port. Needs proper wires / converter.</li>
<li>Using management port RJ-45 (on the back panel). It needs to be configured
however.</li>
<li>Via SSH - but this also needs to be configured properly and you need to have
access credentials.</li>
</ol>
<h1>Connecting using serial/RS-232 port</h1>
<p>This is the method used when a brand new (or brand used :D) switch arrives at
your doorstep. Connecting to this port (which is located on the front panel of
this switch) gives you a direct access to the management console.</p>
<p>It was a daunting thing to do actually. I needed a proper RS-232 -> USB
converter (actually 3rd worked fine, previous two weren't compatible) and also
a proper null-modem cable, that changes order of proper wires (transmit and
receive lines are crosslinked). And lastly - the serial connection from your
workstation to the switch needs to be configured properly.</p>
<p>All of that is an easy thing when you have brand new switch, with all wirings
attached and a setup manual, but in my case it wasn't that easy. But finally I
found a working solution.</p>
<h3>Wiring</h3>
<p>The setup looks like the following:</p>
<p><center><img alt="Extreme Networks Summit X450a-48t wiring setup" src="https://maciej.lasyk.info/images/x450a-48t-wires2.png" width="550px" ></center></p>
<p><center><img alt="Extreme Networks Summit X450a-48t wiring setup" src="https://maciej.lasyk.info/images/x450a-48t-wires.png" width="550px" ></center></p>
<p>So it is:</p>
<ol>
<li>The <a href="https://en.wikipedia.org/wiki/Null_modem">null modem</a> cable (RS-232,
female/female)</li>
<li>The Digitus RS-232 -> USB converter v1.1. Don't use USBv2.0 version(!)</li>
<li>(optionally) USBv1.0 extension cord. It is usually shipped w/above Digitus
converter. It is not required if you can fit the converter into USB port of your workstation directly.</li>
</ol>
<p>Now just assembly & attach everything and boot the switch and wait 2-3 minutes
(sometimes boot process take a moment).</p>
<h3>Serial connection</h3>
<p>I tried this on Linux and Windows, and both connection work using same
parameters. On Linux I used <a href="https://linux.die.net/man/1/minicom">minicom</a>, on
Windows, a basic <a href="https://www.putty.org/">Putty</a>.</p>
<p>Config options:</p>
<ul>
<li><strong>speed / baud rate</strong>: 9600 (manual said 115200, but it didn't work for me)</li>
<li><strong>Data bits</strong>: 8</li>
<li><strong>Stop bits</strong>: 1</li>
<li><strong>Parity</strong>: None</li>
<li><strong>Flow control</strong>: XON/XOFF (manual said to use None flow control, but it
didn't work for me)</li>
</ul>
<p>Remember - after turning on the serial connection, <strong>the screen will go black</strong>.
You need to hit enter couple of times to see the actual console.</p>
<h1>Resetting switch to zero - configuration</h1>
<p>After connecting to the switch, I needed to perform configuration reset.
Normally, there is a CLI command for that (<code>unconfigure switch all</code>), but I
couldn't use it, as I didn't have actuall access to login to the switch (there
was an old system with some old credentials configured).</p>
<p>So the procedure went as following:</p>
<ol>
<li>Power cycle the switch, while connected to the serial console</li>
<li>On the boot prompt, press and hole the spacebar. It will get you to the
boot prompt</li>
<li>At the boot prompt, type command: <strong>config none</strong></li>
<li>Type command: <strong>boot</strong> to boot switch into zero-config mode.</li>
</ol>
<div class="highlight"><pre><span></span>Using<span class="w"> </span>Default<span class="w"> </span>image<span class="w"> </span>...
Press<span class="w"> </span>and<span class="w"> </span>hold<span class="w"> </span>the<span class="w"> </span>to<span class="w"> </span>enter<span class="w"> </span>the<span class="w"> </span>bootrom...
BootRom<span class="w"> </span>><span class="w"> </span>config<span class="w"> </span>none
BootRom<span class="w"> </span>><span class="w"> </span>boot
</pre></div>
<p>Now, after the switch is started with zero-config, login into admin console
(login: <strong>admin</strong>, password is empty, just hit enter). After loggint in, you
need to overwrite old configuration with the new one, by using <strong>save</strong> command:</p>
<div class="highlight"><pre><span></span><span class="o">(</span>pending-AAA<span class="o">)</span><span class="w"> </span>login:
Authentication<span class="w"> </span>Service<span class="w"> </span><span class="o">(</span>AAA<span class="o">)</span><span class="w"> </span>on<span class="w"> </span>the<span class="w"> </span>master<span class="w"> </span>node<span class="w"> </span>is<span class="w"> </span>now<span class="w"> </span>available<span class="w"> </span><span class="k">for</span><span class="w"> </span>login.
login:<span class="w"> </span>admin
password:
ExtremeXOS
Copyright<span class="w"> </span><span class="o">(</span>C<span class="o">)</span><span class="w"> </span><span class="m">1996</span>-2016<span class="w"> </span>Extreme<span class="w"> </span>Networks.<span class="w"> </span>All<span class="w"> </span>rights<span class="w"> </span>reserved.
<span class="o">[</span>...<span class="o">]</span>
*<span class="w"> </span>Summit<span class="w"> </span><span class="c1"># save</span>
The<span class="w"> </span>configuration<span class="w"> </span>file<span class="w"> </span>primary.cfg<span class="w"> </span>already<span class="w"> </span>exists.
Do<span class="w"> </span>you<span class="w"> </span>want<span class="w"> </span>to<span class="w"> </span>save<span class="w"> </span>configuration<span class="w"> </span>to<span class="w"> </span>primary.cfg<span class="w"> </span>and<span class="w"> </span>overwrite<span class="w"> </span>it?<span class="w"> </span><span class="o">(</span>y/N<span class="o">)</span><span class="w"> </span>Yes
Saving<span class="w"> </span>configuration<span class="w"> </span>on<span class="w"> </span>master<span class="w"> </span>.........<span class="w"> </span><span class="k">done</span>!
Configuration<span class="w"> </span>saved<span class="w"> </span>to<span class="w"> </span>primary.cfg<span class="w"> </span>successfully.
</pre></div>
<p>And everything is done, your brand-almost-new switch is ready!</p>Problems with imports using bitnami/mariadb2022-05-01T00:00:00+02:002022-05-01T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2022-05-01:2022/May/01/problems-with-imports-using-bitnamimariadb/<p><center><img alt="Bitnami MariaDB" src="https://maciej.lasyk.info/images/mariadb-logo.png" width="550px" ></center></p>
<h1>What is it about?</h1>
<p>Lately, while trying to import a Mariadb backup into database (hosted on
Kubernetes and using <a href="https://github.com/bitnami/charts/tree/master/bitnami/mariadb">Bitnami Mariadb</a>)
I was hitting issues like following:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>mysql<span class="w"> </span>-u<span class="w"> </span>root<span class="w"> </span>-h<span class="w"> </span><edited-ip><span class="w"> </span>-p<span class="w"> </span><edited-dbname><span class="w"> </span><<span class="w"> </span>backup.sql
Enter<span class="w"> </span>password:
ERROR<span class="w"> </span><span class="m">2013</span><span class="w"> </span><span class="o">(</span>HY000<span class="o">)</span><span class="w"> </span>at<span class="w"> </span>line<span class="w"> </span><span class="m">1302</span>:<span class="w"> </span>Lost<span class="w"> </span>connection<span class="w"> </span>to<span class="w"> </span>MySQL<span class="w"> </span>server<span class="w"> </span>during<span class="w"> </span>query
</pre></div>
<h1>Investigation</h1>
<p>So, after I started the process of import, it was disrupted. I have verified
Mariadb server, which was running in a k8s pod, and noticed it was restarted.</p>
<p>A mariadb restart would definitely cause above client error ("Lost connection
to MySQL server during query").</p>
<p>Question is - why was it restarted? So I dug into k8s events and saw that <strong>
Liveness probe failed</strong>.</p>
<p>Liveness probe in this Bitnami/Mariadb chart is a simple <a href="https://github.com/bitnami/charts/blob/master/bitnami/mariadb/templates/primary/statefulset.yaml#L214">mysqladmin command</a>:</p>
<div class="highlight"><pre><span></span><span class="w"> </span>livenessProbe:<span class="w"> </span><span class="o">{{</span>-<span class="w"> </span>omit<span class="w"> </span>.Values.primary.livenessProbe<span class="w"> </span><span class="s2">"enabled"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>toYaml<span class="w"> </span><span class="p">|</span><span class="w"> </span>nindent<span class="w"> </span><span class="m">12</span><span class="w"> </span><span class="o">}}</span>
<span class="w"> </span>exec:
<span class="w"> </span>command:
<span class="w"> </span>-<span class="w"> </span>/bin/bash
<span class="w"> </span>-<span class="w"> </span>-ec
<span class="w"> </span>-<span class="w"> </span><span class="p">|</span>
<span class="w"> </span><span class="nv">password_aux</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">MARIADB_ROOT_PASSWORD</span><span class="k">:-</span><span class="si">}</span><span class="s2">"</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[[</span><span class="w"> </span>-f<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">MARIADB_ROOT_PASSWORD_FILE</span><span class="k">:-</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="o">]]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nv">password_aux</span><span class="o">=</span><span class="k">$(</span>cat<span class="w"> </span><span class="s2">"</span><span class="nv">$MARIADB_ROOT_PASSWORD_FILE</span><span class="s2">"</span><span class="k">)</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span>mysqladmin<span class="w"> </span>status<span class="w"> </span>-uroot<span class="w"> </span>-p<span class="s2">"</span><span class="si">${</span><span class="nv">password_aux</span><span class="si">}</span><span class="s2">"</span>
<span class="w"> </span><span class="o">{{</span>-<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span>.Values.primary.customLivenessProbe<span class="w"> </span><span class="o">}}</span>
<span class="w"> </span>livenessProbe:<span class="w"> </span><span class="o">{{</span>-<span class="w"> </span>include<span class="w"> </span><span class="s2">"common.tplvalues.render"</span><span class="w"> </span><span class="o">(</span>dict<span class="w"> </span><span class="s2">"value"</span><span class="w"> </span>.Values.primary.customLivenessProbe<span class="w"> </span><span class="s2">"context"</span><span class="w"> </span>$<span class="o">)</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>nindent<span class="w"> </span><span class="m">12</span><span class="w"> </span><span class="o">}}</span>
<span class="w"> </span><span class="o">{{</span>-<span class="w"> </span>end<span class="w"> </span><span class="o">}}</span>
<span class="w"> </span><span class="o">{{</span>-<span class="w"> </span><span class="k">if</span><span class="w"> </span>.Values.primary.readinessProbe.enabled<span class="w"> </span><span class="o">}}</span>
</pre></div>
<p>Why would it stop working during an import process?</p>
<h1>Resolution</h1>
<p>The short answer is: import process creates quite huge performance pressure.
Mariadb pod stops responding for anything, while this import is in progress.
That's because I'm running a low - power, thin-client devices as k8s nodes.</p>
<p>So - livenessprobe fails, and k8s scheduler restarts pod, and everything gets
back to normal. But database import is not finished and needs to be started all
over again.</p>
<p>Solution was to actually change default timeouts in liveness probes for the time
if the import process:</p>
<div class="highlight"><pre><span></span><span class="w"> </span>livenessProbe:
<span class="w"> </span>enabled:<span class="w"> </span><span class="nb">true</span>
<span class="w"> </span>initialDelaySeconds:<span class="w"> </span><span class="m">120</span>
<span class="w"> </span>periodSeconds:<span class="w"> </span><span class="m">30</span>
<span class="w"> </span>timeoutSeconds:<span class="w"> </span><span class="m">120</span>
<span class="w"> </span>failureThreshold:<span class="w"> </span><span class="m">30</span>
<span class="w"> </span>successThreshold:<span class="w"> </span><span class="m">1</span>
</pre></div>
<p>And that was it. After import was successfull, I reverted above values to
defaults.</p>CoreDNS problems with Mikrotik router2022-04-12T00:00:00+02:002022-04-12T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2022-04-12:2022/Apr/12/coredns-problems-with-mikrotik-router/<p><center><img alt="Mikrotik" src="https://maciej.lasyk.info/images/mt.png" width="550px" ></center></p>
<h2>Problem?</h2>
<p>When working on Elastic/Prometheus integration (long-term storage) I noticed
weird problems with DNS resolution. Namely, Prometheus pod was not able to
resolve my Elastic Cloud instance hostname.</p>
<p>Verified with CoreDNS graphs, that there is something weird going on:</p>
<p><center><img alt="CoreDNS" src="https://maciej.lasyk.info/images/coredns-problem-1.png" width="550px" ></center></p>
<p>So dug into coredns/nodelocaldns DaemonSets pods logs, and found following problems:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>ERROR<span class="o">]</span><span class="w"> </span>plugin/errors:<span class="w"> </span><span class="m">2</span><span class="w"> </span>netrunner.es.eu-central-1.aws.cloud.es.io.<span class="w"> </span>A:<span class="w"> </span>dns:<span class="w"> </span>overflow<span class="w"> </span>unpacking<span class="w"> </span>uint32
<span class="o">[</span>ERROR<span class="o">]</span><span class="w"> </span>plugin/errors:<span class="w"> </span><span class="m">2</span><span class="w"> </span>netrunner.es.eu-central-1.aws.cloud.es.io.<span class="w"> </span>A:<span class="w"> </span>dns:<span class="w"> </span>overflowing<span class="w"> </span>header<span class="w"> </span>size
<span class="o">[</span>ERROR<span class="o">]</span><span class="w"> </span>plugin/errors:<span class="w"> </span><span class="m">2</span><span class="w"> </span>netrunner.es.eu-central-1.aws.cloud.es.io.<span class="w"> </span>A:<span class="w"> </span>dns:<span class="w"> </span>buffer<span class="w"> </span>size<span class="w"> </span>too<span class="w"> </span>small
</pre></div>
<p>Wow, weird. I quickly verified w/tcpdump and wireshark, that indeed DNS payload
in UDP dataframes getting back from my DNS server (<a href="https://mikrotik.com/product/RB750Gr3">Mikrotik RouterBoard RB750Gr3</a>) were much bigger for this query.</p>
<p>DNS has this limit of 512 bytes for UDP for a reason (it guarantees that DNS
packets can be reassembled if fragmented in transit). For larger payloads TCP
is used.</p>
<p>So in my case, looks like DNS server returned a bigger DNS UDP payload than
RFC allows. And CoreDNS didn't really like it.</p>
<h2>Solution?</h2>
<p>I quickly took a look at <a href="https://wiki.mikrotik.com/wiki/Manual:IP/DNS">router configuration options</a> (RouterOS) referring to DNS, and quickly realized, that by default:</p>
<p><strong>max-udp-packet-size (integer [50..65507]; Default: 4096) Maximum size of allowed UDP packet.</strong></p>
<p>Verified this w/my router configuration and that was it. The solution was to:</p>
<div class="highlight"><pre><span></span>/ip<span class="w"> </span>dns<span class="w"> </span><span class="nb">set</span><span class="w"> </span>max-udp-packet-size<span class="o">=</span><span class="m">512</span>
</pre></div>Testing MariaDB/Mysql backups restoration in Kubernetes2022-03-17T00:00:00+01:002022-03-17T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2022-03-17:2022/Mar/17/testing-mariadbmysql-backups-restoration-in-kubernetes/<p><center><img alt="MySQL logo" src="https://maciej.lasyk.info/images/mysql_logo.png" width="550px" ></center></p>
<h2>What is it about?</h2>
<p>Shortly - how to test Mysql/Mariadb backups in an automated manner in Kubernetes? This article briefly describes, how to do it.</p>
<h2>Why?</h2>
<p>There're still many people, who hosts own databases servers. Even using K8S (yeah, I know - this might be an antipattern, but not always. I's an academic discussion imo).</p>
<p>So, having your own DB server, you need to take care of taking backups. And recovering data, when there's a need. And you can only tell, that you've got backups covered only (and really - ONLY), when those backups are tested.</p>
<h2>Testing backup file - the procedure</h2>
<ol>
<li>Fetch the backup file from remote location</li>
<li>Import it to the new/vanilla Mysql/MariaDB server</li>
<li>Run integrity tests (e.g. CHECK TABLE) on each table, that was recovered</li>
</ol>
<h2>Putting it into Kubernetes</h2>
<p>The K8S object would be Cronjob of course. I thought about it a bit, and came to conclusion, that the following assumptions should be correct for this workload:</p>
<ol>
<li>Start Mysql/MariaDB container from an upstream image (e.g. <a href="https://hub.docker.com/_/mariadb">MariaDB image</a>)</li>
<li>Create another container (using custom image), that:<ol>
<li>Fetches the backup image from remote</li>
<li>Unarchives this backup archive</li>
<li>Imports the data into Mysql/MariaDB server container runing in the same pod</li>
<li>Runs CHECK TABLE SQL commands to verify data integrity</li>
<li>Send SHUTDOWN command to Mysql/MariaDB container, so this container exits with 0 - exit code (SUCCESS)</li>
</ol>
</li>
<li>If in any point of this procedure anything fails - the whole Cronjob fails. And Kubernetes monitoring (e.g. Prometheus) may easily alert on that.</li>
</ol>
<p>And that's it. This way the test environment is always fresh, no data is left, disk space is recovered and that should be enough.</p>
<h2>Show me the code!</h2>
<p>Ok, ok - BUT! Don't judge. I wrote it just for myself, for my internal use => it's not a pretty code. But it's enough for the job. You may rewrite it in any language of your choice - that's obvious.</p>
<ol>
<li>The container which run tests:<ol>
<li><a href="https://maciej.lasyk.info/static/2022-03-17/Containerfile.txt">Containerfile</a> which defines the "test" container image</li>
<li><a href="https://maciej.lasyk.info/static/2022-03-17/gsutil.repo.txt">gsutil.repo used by this container</a></li>
<li><a href="https://maciej.lasyk.info/static/2022-03-17/entrypoint.sh.txt">entrypoint.sh used by this container</a></li>
</ol>
</li>
<li>The <a href="https://maciej.lasyk.info/static/2022-03-17/Cronjob.yaml.txt">Cronjob resource</a></li>
</ol>
<h2>What's missing here?</h2>
<p>My experience tells me, that there's one more thing needed to be checked in terms of database backups. The size of the backup. Sometimes it happens, that data volume shrinks for some reason. Often those kinds of situations should be verified. Above article doesn't really cover this matter. That's mainly because imo this kind of test is hard to be done by this kind of Cronjob in a situation, when Prometheus is used for monitoring.</p>
<p>Prometheus is pull-based, meaning, there needs to be some service, which provides metrics to it. Described above Cronjob doesn't really provide an API serving any data to Prometheus.</p>
<p>There are several ways to solve this kind of problem. But hey - I'm leaving this to you, as it heavily depends on your monitoring setup.</p>A curious case of MariaDB on Longhorn/K8S2022-01-14T00:00:00+01:002022-01-14T00:00:00+01:00Maciejtag:maciej.lasyk.info,2022-01-14:2022/Jan/14/a-curious-case-of-mariadb-on-longhornk8s/<p><center><img alt="K8S logo" src="https://maciej.lasyk.info/images/k8s-logo.png" width="550px" ></center></p>
<h2>What is it about?</h2>
<p>In my home (not "lab", but "prod" XD) datacenter (ok, a basement one)
I have this HA setup of MariaDB (by <a href="https://github.com/bitnami/charts/tree/master/bitnami/mariadb">Bitnami</a>]).</p>
<p>The topology is very simple (inside of Kubernetes, so "let's say simple"):</p>
<ol>
<li>Primary node is hosted on top of <a href="https://longhorn.io/">Longhorn volume</a> - thanks to that there is no network communication between mariadb process and its storage, so no latency. Also, volume is a local filesystem mount, so any kind of file-locking operations are available. This is cool, as this Longhorn volume is a distributed, network storage, but from application perspective - it's a local filesystem.</li>
<li>Secondary node is hosted on top of QNAP/NFS - it's a bit slower, but its fine. Storage is very resilient, and also I can afford taking snapshots and mysqldump backups here.</li>
</ol>
<h2>So what is the problem?</h2>
<p>A couple of days ago I needed to perform a rolling restart of all K8S nodes.
All was fine, but shortly afterwards I realised, that MariaDB was constantly crashing due to filesystem errors (a various ones). One of those:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>ERROR<span class="o">]</span><span class="w"> </span><span class="o">[</span>FATAL<span class="o">]</span><span class="w"> </span>InnoDB:<span class="w"> </span>fdatasync<span class="o">()</span><span class="w"> </span>returned<span class="w"> </span><span class="m">5</span>
<span class="o">[</span>ERROR<span class="o">]</span><span class="w"> </span>mysqld<span class="w"> </span>got<span class="w"> </span>signal<span class="w"> </span><span class="m">6</span>
This<span class="w"> </span>could<span class="w"> </span>be<span class="w"> </span>because<span class="w"> </span>you<span class="w"> </span>hit<span class="w"> </span>a<span class="w"> </span>bug.<span class="w"> </span>It<span class="w"> </span>is<span class="w"> </span>also<span class="w"> </span>possible
that<span class="w"> </span>this<span class="w"> </span>binary<span class="w"> </span>or<span class="w"> </span>one<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>libraries<span class="w"> </span>it<span class="w"> </span>was<span class="w"> </span>linked
against<span class="w"> </span>is<span class="w"> </span>corrupt,<span class="w"> </span>improperly<span class="w"> </span>built,<span class="w"> </span>or<span class="w"> </span>misconfigured.
This<span class="w"> </span>error<span class="w"> </span>can<span class="w"> </span>also<span class="w"> </span>be<span class="w"> </span>caused<span class="w"> </span>by<span class="w"> </span>malfunctioning<span class="w"> </span>hardware.
To<span class="w"> </span>report<span class="w"> </span>this<span class="w"> </span>bug,<span class="w"> </span>see<span class="w"> </span>https://mariadb.com/kb/en/reporting-bugs
We<span class="w"> </span>will<span class="w"> </span>try<span class="w"> </span>our<span class="w"> </span>best<span class="w"> </span>to<span class="w"> </span>scrape<span class="w"> </span>up<span class="w"> </span>some<span class="w"> </span>info<span class="w"> </span>that<span class="w"> </span>will
hopefully<span class="w"> </span><span class="nb">help</span><span class="w"> </span>diagnose<span class="w"> </span>the<span class="w"> </span>problem,<span class="w"> </span>but<span class="w"> </span>since<span class="w"> </span>we<span class="w"> </span>have
already<span class="w"> </span>crashed,<span class="w"> </span>something<span class="w"> </span>is<span class="w"> </span>definitely<span class="w"> </span>wrong<span class="w"> </span>and<span class="w"> </span>this
may<span class="w"> </span>fail.
</pre></div>
<p>And I couldn't find the root cause. I performed hardware analysis, run disk repairs (fsck) and verified in dmesg, that there were no actual hardware problems. Still no luck.</p>
<h2>Closer to solution</h2>
<p>I started thinking about Longhorn itself. It was running perfectly in this setup for last 2 months, but maybe something was off here.</p>
<p>So I looked into logs of <strong>instance-manager</strong> (a Longhorn component hosted on each node that provides volumes to workloads) and found this, a seconds before yet-another-mariadb-crash:</p>
<div class="highlight"><pre><span></span><span class="nv">time</span><span class="o">=</span><span class="s2">"2022-01-14T12:45:51Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>debug<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Stop monitoring instance manager instance-manager-e-ed9b8934"</span><span class="w"> </span><span class="nv">controller</span><span class="o">=</span>longhorn-instance-manager<span class="w"> </span>instance<span class="w"> </span><span class="nv">manager</span><span class="o">=</span>instance-manager-e-ed9b8934<span class="w"> </span><span class="nv">node</span><span class="o">=</span>node6
<span class="nv">time</span><span class="o">=</span><span class="s2">"2022-01-14T12:45:51Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>error<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"error receiving next item in engine watch: rpc error: code = Canceled desc = grpc: the client connection is closing"</span><span class="w"> </span><span class="nv">controller</span><span class="o">=</span>longhorn-instance-manager<span class="w"> </span>instance<span class="w"> </span><span class="nv">manager</span><span class="o">=</span>instance-manager-e-ed9b8934<span class="w"> </span><span class="nv">node</span><span class="o">=</span>node6
<span class="nv">time</span><span class="o">=</span><span class="s2">"2022-01-14T12:45:51Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>debug<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"removed the engine from imc.instanceManagerMonitorMap"</span><span class="w"> </span><span class="nv">controller</span><span class="o">=</span>longhorn-instance-manager<span class="w"> </span>instance<span class="w"> </span><span class="nv">manager</span><span class="o">=</span>instance-manager-e-ed9b8934<span class="w"> </span><span class="nv">node</span><span class="o">=</span>node6
<span class="nv">time</span><span class="o">=</span><span class="s2">"2022-01-14T12:45:51Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>warning<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Cannot find the instance manager for the running instance pvc-a66176a1-f874-4974-a53c-28e69578dc32-e-b41f41ca, will mark the instance as state ERROR"</span>
<span class="nv">time</span><span class="o">=</span><span class="s2">"2022-01-14T12:45:51Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>debug<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Instance handler updated instance pvc-a66176a1-f874-4974-a53c-28e69578dc32-e-b41f41ca state, old state running, new state error"</span>
<span class="nv">time</span><span class="o">=</span><span class="s2">"2022-01-14T12:45:51Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>debug<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Requeue volume due to error <nil> or Operation cannot be fulfilled on engines.longhorn.io \"pvc-a66176a1-f874-4974-a53c-28e69578dc32-e-b41f41ca\": the object has been modified; please apply your changes to the latest version and try again"</span><span class="w"> </span><span class="nv">accessMode</span><span class="o">=</span>rwo<span class="w"> </span><span class="nv">controller</span><span class="o">=</span>longhorn-volume<span class="w"> </span><span class="nv">frontend</span><span class="o">=</span>blockdev<span class="w"> </span><span class="nv">migratable</span><span class="o">=</span><span class="nb">false</span><span class="w"> </span><span class="nv">node</span><span class="o">=</span>node6<span class="w"> </span><span class="nv">owner</span><span class="o">=</span>node6<span class="w"> </span><span class="nv">state</span><span class="o">=</span>attached<span class="w"> </span><span class="nv">volume</span><span class="o">=</span>pvc-a66176a1-f874-4974-a53c-28e69578dc32
<span class="nv">time</span><span class="o">=</span><span class="s2">"2022-01-14T12:45:51Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>warning<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Engine of volume dead unexpectedly, reattach the volume"</span><span class="w"> </span><span class="nv">accessMode</span><span class="o">=</span>rwo<span class="w"> </span><span class="nv">controller</span><span class="o">=</span>longhorn-volume<span class="w"> </span><span class="nv">frontend</span><span class="o">=</span>blockdev<span class="w"> </span><span class="nv">migratable</span><span class="o">=</span><span class="nb">false</span><span class="w"> </span><span class="nv">node</span><span class="o">=</span>node6<span class="w"> </span><span class="nv">owner</span><span class="o">=</span>node6<span class="w"> </span><span class="nv">state</span><span class="o">=</span>attached<span class="w"> </span><span class="nv">volume</span><span class="o">=</span>pvc-a66176a1-f874-4974-a53c-28e69578dc32
<span class="nv">time</span><span class="o">=</span><span class="s2">"2022-01-14T12:45:51Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>info<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-a66176a1-f874-4974-a53c-28e69578dc32\", UID:\"d00b2f85-a6ad-494b-ac81-906b4b01ce31\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"15833564\", FieldPath:\"\"}): type: 'Warning' reason: 'DetachedUnexpectly' Engine of volume pvc-a66176a1-f874-4974-a53c-28e69578dc32 dead unexpectedly, reattach the volume"</span>
<span class="nv">time</span><span class="o">=</span><span class="s2">"2022-01-14T12:45:51Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>debug<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Stop monitoring engine"</span><span class="w"> </span><span class="nv">controller</span><span class="o">=</span>longhorn-engine<span class="w"> </span><span class="nv">engine</span><span class="o">=</span>pvc-a66176a1-f874-4974-a53c-28e69578dc32-e-b41f41ca<span class="w"> </span><span class="nv">node</span><span class="o">=</span>node6
</pre></div>
<p>Bingo! It is something odd with Longhorn itself. I confirmed, that problem was: <strong>instance-manager pods were restarting every 60 minutes</strong>. A very weird problem.</p>
<h2>Even closer to solution</h2>
<p>So the question now was: "why this started happening shortly after node were rebooted?".</p>
<p>I even reinstalled whole Longhorn in the meantime, just to rule-out some manual changes, that I might have done in the meantime.</p>
<p>I scrolled through git log of changes in Longhorn deployments, and quickly found out, that 2 weeks ago I added <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/">priorityClasses</a> to Longhorn components - just to make sure no volume PODs are ever evicted.</p>
<p>So I verified those priorityClasses and found out, that <strong>for some reason, instance-manager PODs have default priorityClass</strong>. Wow.</p>
<p>Not sure why, as I <a href="https://longhorn.github.io/longhorn-tests/manual/release-specific/v1.1.1/tolerations_priorityclass_setting/">configured priorityClasses correctly</a> in my Helm charts and it was fine before nodes were rebooted.</p>
<h2>Solution?</h2>
<p>Basically make sure, all Longhorn components have some high-level priority class. I created a custom one (called high-ppriority :) ), with a very high index.</p>
<p>And verify it:</p>
<div class="highlight"><pre><span></span>kubectl<span class="w"> </span>-n<span class="w"> </span>longhorn-system<span class="w"> </span>describe<span class="w"> </span>pods<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'Priority Class'</span>
</pre></div>
<h2>PS</h2>
<p>I even found a bug request for that: <a href="https://github.com/longhorn/longhorn/issues/2820">https://github.com/longhorn/longhorn/issues/2820</a></p>ipcalc-contains2022-01-03T00:00:00+01:002022-01-03T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2022-01-03:2022/Jan/03/ipcalc-contains/<p><center><img alt="ipcalc-contains" src="https://maciej.lasyk.info/images/ipcalc-contains.png" /></center></p>
<h2>Problem?</h2>
<p>I was doing some cloud - related networking job. I had to verify whether couple of networks contains some IP addresses.</p>
<p>In a situation, when there's only a couple of checks to be done, writing a script doesn't make a sense. In such circumstances I usually use <a href="https://linux.die.net/man/1/ipcalc">ipcalc</a> binary.</p>
<p>However this binary requires from me "a manual, eye - based verification". I basically need to verify by myself it a network, which is in details explained by ipcalc contains a specific IP address or not.</p>
<p>As this verification is very easy in any high - level programming language, thus I decided to create a lightweight binary for that in Golang.</p>
<p>Additional bonus - in latest version you can also verify if a network contains another network.</p>
<h2>Where is it?</h2>
<p>Here: <a href="https://github.com/docent-net/ipcalc-contains">https://github.com/docent-net/ipcalc-contains</a></p>
<p>You can download binaries from here: <a href="https://github.com/docent-net/ipcalc-contains/releases">https://github.com/docent-net/ipcalc-contains/releases</a></p>
<h2>How to use it?</h2>
<p>Simply <a href="https://github.com/docent-net/ipcalc-contains/releases">download a library that fits your OS/architecture</a> and put inside of your ~/bin directory.</p>Replacing provider in Terraform2021-01-12T00:00:00+01:002021-01-12T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2021-01-12:2021/Jan/12/replacing-provider-in-terraform/<h2>Problem?</h2>
<p>Lately, during upgrading Terraform code to 0.13.x I spotted an error:</p>
<div class="highlight"><pre><span></span><span class="n">Error</span><span class="o">:</span><span class="w"> </span><span class="n">Provider</span><span class="w"> </span><span class="n">configuration</span><span class="w"> </span><span class="n">not</span><span class="w"> </span><span class="n">present</span>
<span class="n">To</span><span class="w"> </span><span class="n">work</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">vault_kubernetes_auth_backend_role</span><span class="o">.</span><span class="na">some_role</span>
<span class="n">its</span><span class="w"> </span><span class="n">original</span><span class="w"> </span><span class="n">provider</span><span class="w"> </span><span class="n">configuration</span><span class="w"> </span><span class="n">at</span>
<span class="n">provider</span><span class="o">[</span><span class="s2">"registry.terraform.io/-/vault"</span><span class="o">]</span>
<span class="k">is</span><span class="w"> </span><span class="n">required</span><span class="o">,</span><span class="w"> </span><span class="n">but</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="n">has</span><span class="w"> </span><span class="n">been</span><span class="w"> </span><span class="n">removed</span><span class="o">.</span><span class="w"> </span><span class="n">This</span><span class="w"> </span><span class="n">occurs</span><span class="w"> </span><span class="n">when</span>
<span class="n">a</span><span class="w"> </span><span class="n">provider</span><span class="w"> </span><span class="n">configuration</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">removed</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">objects</span><span class="w"> </span><span class="n">created</span>
<span class="n">by</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">provider</span><span class="w"> </span><span class="n">still</span><span class="w"> </span><span class="n">exist</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">state</span><span class="o">.</span><span class="w"> </span><span class="n">Re</span><span class="o">-</span><span class="n">add</span><span class="w"> </span><span class="n">the</span>
<span class="n">provider</span><span class="w"> </span><span class="n">configuration</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">destroy</span>
<span class="n">vault_kubernetes_auth_backend_role</span><span class="o">.</span><span class="na">some_role</span><span class="o">,</span><span class="w"> </span><span class="n">after</span>
<span class="n">which</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="n">remove</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">provider</span><span class="w"> </span><span class="n">configuration</span><span class="w"> </span><span class="n">again</span><span class="o">.</span>
</pre></div>
<h2>Why?</h2>
<p>So, this problem is actually covered in <a href="https://www.terraform.io/upgrade-guides/0-13.html#explicit-provider-source-locations">Upgrading to Terraform v0.13 / Explicit Provider Source Locations</a> docs:</p>
<p>Basically, URLs for providers changed (imo for better, namespaced providers created by community now will be easier to maintain).</p>
<p>Above error wouldn't happen if I didn't replace the provider declaration in
my main.tf with the following:</p>
<div class="highlight"><pre><span></span>terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "1.13.3"
}
}
}
</pre></div>
<h2>Solution?</h2>
<p>We need to replace the provider URL in the state file. So:</p>
<ol>
<li>Download your state file to local drive (e.g. from S3, Azure Blob or where you keep it) and create a backup copy just to be sure</li>
<li>Use command (adjust to your provider): <code>terraform state replace-provider -state=<TFSTATE-FILENAME> registry.terraform.io/-/kubernetes registry.terraform.io/hashicorp/kubernetes</code></li>
<li>Upload the state file back to your remote storage</li>
<li>Run <code>terraform plan</code> to just make sure everything works fine now</li>
</ol>
<p>This should be enough. And remember to keep S3 bucket versioned (if using S3 as backend for state files), because working directly on state files this way might be dangerous.</p>Site Reliability Engineering talks2019-06-17T00:00:00+02:002019-06-17T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2019-06-17:2019/Jun/17/site-reliability-engineering-talks/<p><center><img alt="Kielce!" src="https://maciej.lasyk.info/images/maciek-sre.jpg" /></center></p>
<p>Some time ago a friend of mine, who is a co - organizer of <a href="https://geecon.org/">Geecon</a>
(thanks <a href="https://twitter.com/ags313">Ags</a>!) asked me about giving a
talk about "how to maintain applications on production".</p>
<p>So I thought that instead of some generic talk I'll work on a presentation
where I would describe
in details how we worked on reliability matters @<a href="https://codewise.com/">Codewise</a>.
And the story should be interesting to people, as our company is well known for
running low - latency, high traffic systems (400k requests/s total).</p>
<p><center><img alt="Poznań!" src="https://maciej.lasyk.info/images/maciek-devopsdays-poznan.jpg" /></center></p>
<p>I run this presentation three times, second even as a keynote during
<a href="https://devopsdays.org/events/2019-poznan/welcome/">DevopsDays Poznań</a> event.</p>
<p>The interesting part is originally this presentation was meant to tell a story
of a no-ops company running own Cassandra clusters, however after Geecon
edition I changed it a bit and removed Cassandra - related slides, as it turned
out that people are very interested in reliability subjects but in a generic
approach (not focused only on one service).</p>
<p>So here are slides (videos are still not available): <a href="https://github.com/docent-net/conferences/blob/master/2019-devopsdays-cassandra-sre-implements-devops/cassandra-sre-implements-devops-devopsdays.pdf">click</a></p>Recap from 16th Linux Session @Wrocław2019-04-07T00:00:00+02:002019-04-07T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2019-04-07:2019/Apr/07/recap-from-16th-linux-session-wroclaw/<p><center><img alt="16th Linux Session!" src="https://maciej.lasyk.info/images/16-sesja-logo.png" /></center></p>
<p><a href="https://sesja.linuksowa.pl">16th Linux Session</a> @Wrocław is a wrap and I must say that it was one of the best editions
from my point of view. I could finally stay for the whole weekend, enjoy discussions with most of attendees and learn
something new. And have a beer or two or more ;)</p>
<p>From all those talks I attended in I'd like to say that two draw my attention quite close:</p>
<ol>
<li><strong>Performant filtering of network packets with BPF</strong> by <a href="https://twitter.com/mrostecki?lang=en">Michał Rostecki</a>.
Michał works around <a href="https://cilium.io/">Cilium</a> at <a href="https://www.suse.com/">Suse</a> and during his talk I learnt a bit
about how current state of network filtering works in Linux Kernel, what are different methods of filtering / routing
and where to find one of the best documentation about BPF (that is on Cilium documentation pages: <a href="https://cilium.readthedocs.io/en/latest/bpf/">https://cilium.readthedocs.io/en/latest/bpf/</a>).</li>
<li><strong>Own email server - hows and whys</strong> by Błażej Święcicki (Błażej, do you exist on internets at all? ;) ). This talk
started interesting discussion about privacy of our communication. We learnt how it looks like to host own email
service these days. Personally I do host my own mail server thus I was interested in challenges Błażej had to accept
during last decade of maintaining Postfixes and making sure emails are properly delivered and received.</li>
</ol>
<p>I was a speaker at this Linux Session for the 6th time - it's actually the only conference I attend so regularly -
thanks guys! So this time I talked about security of Linux containers (one more time). Slides are here:</p>
<p><center><a href="http://maciej.lasyk.info/slides/16th-linux-session-slides.pdf"><img alt="16th Linux Session!" src="https://maciej.lasyk.info/images/16th-linux-session-slides.png" /></a>
<a href="http://maciej.lasyk.info/slides/16th-linux-session-slides.pdf">16th-linux-session-slides.pdf</a>
</center></p>
<p>Also I decided to prepare a lightning - talks. First about <a href="https://github.com/sshuttle/sshuttle">sshuttle</a> and the
second one about my hobby which is a long - distance training and sport and how Linux
and open source software helps me with that. 2 years ago I finished <a href="https://nxtri.com/">Norseman Xtreme Triahlon</a> -
during this lightning talk I described briefly how I use <a href="https://www.goldencheetah.org">GoldenCheetah</a> for my workout
performance analysis and planning, <a href="https://www.blender.org/">Blender</a> for video analysis of my swimming stroke and
<a href="https://www.audacityteam.org/">Audacity</a> for audio editing and working on content of my
<a href="https://dlugodystansowy.pl">sport - related podcast</a>.</p>
<p>Slides from this lightning talk are here: <a href="https://maciej.lasyk.info/slides/16th-linux-session-lt">https://maciej.lasyk.info/slides/16th-linux-session-lt</a> </p>
<p>Last but not least - some time ago one of <a href="https://twitter.com/benbreardjr/status/1113122811042848768">tweets</a> draw my attention:</p>
<p><center><img alt="systemd ftw" src="https://maciej.lasyk.info/images/ben-systemd.png" /></center></p>
<p>I liked this t-shirt very much, and as I always try to teach more and more people about good things of systemd therefore
I decided to prepare similar t-shirt for this event. Here it is (you can't really see the sentence, but believe me -
it's there ;) ):</p>
<p><center><img alt="systemd ftw" src="https://maciej.lasyk.info/images/everybody-loves-systemd.jpg" /></center></p>
<p>Thank you guys and see you next year!</p>More about containers security2019-03-01T00:00:00+01:002019-03-01T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2019-03-01:2019/Mar/01/more-about-containers-security/<p><center><img alt="OWASP!" src="https://maciej.lasyk.info/images/OWASP_Poland_logo.png" /></center></p>
<p>Two days before this event <a href="https://pl.linkedin.com/in/wojciechdworakowski">Wojciech Dworakowki</a>
(leader of <a href="https://www.owasp.org/index.php/Poland">Polish OWASP chapter</a>)
asked me if I could prepare an emergency presentation on a very short
notice as one of already announced and confirmed speakers lost his voice due to
a sickness. As I always try to support OWASP and have some practical knowledge
and experience related to infrastructure security I agreed. It cost me only one
evening / 5h to prepare this presentation.</p>
<p>I must write, that I was positively surprised as I wasn't completely sure about
quality of slides, content and my pronunciation as I didn't have time at all
to practice (and I'm not a native, English speaker). Fortunately feedback I
received was very positive, so thanks guys!</p>
<p>I published slides here (uninfected PDF for downloading): </p>
<p><center><a href="http://maciej.lasyk.info/slides/OWASP_more_about_linux_containers.pdf"><img alt="OWASP!" src="https://maciej.lasyk.info/images/owasp-more-about-containers-security.png" /></a>
<a href="http://maciej.lasyk.info/slides/OWASP_more_about_linux_containers.pdf">OWASP_more_about_linux_containers.pdf</a>
</center></p>
<p>By the way - I did it again - 2 years ago I gave an emergency talk at huge
conference (Devopsdays Poland) on a 1,5h notice(!). You can view it
<a href="https://www.youtube.com/watch?v=KPB51TmsUEY">here</a> - It also was far from
ideal, but again - back then feedback was also good. During that time I was a
member of the conference program committee and basically one of speakers didn't
appear and I had to replace him. I remember that I had to find a calm place
and having only 90 minutes prepare a 25min talk. Crazy stuff :D </p>
<p>So - funny stories, that tell me that you never know the hour, so stay always
prepared. And public speaking is fun! ;)</p>
<p>Enjoy!</p>Coredump conference talk2019-02-16T00:00:00+01:002019-02-16T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2019-02-16:2019/Feb/16/coredump-conference-talk/<p><center><img alt="Coredump conference" src="https://maciej.lasyk.info/images/coredump-logo.png" /></center></p>
<p>Last year together with <a href="https://twitter.com/dmnlukasik">Damian Łukasik</a> we
gave a talk about configuration management using Spring Cloud Config and some
Python / Rundeck / BASH glue for High Available deployment across AWS and
Google Cloud. It was a case study from our work at Codewise.</p>
<p>This was a part of <a href="http://coredump.events/2018/">Coredump conference</a> agenda.</p>
<p>You can watch video here:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/fKCumCt_4sE" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Enjoy!</p>Devopsdays Warsaw one more time2019-02-15T00:00:00+01:002019-02-15T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2019-02-15:2019/Feb/15/devopsdays-warsaw-one-more-time/<p><center><img alt="Devopsdays!" src="https://maciej.lasyk.info/images/DevOpsDays.png" /></center></p>
<p>Last year I gave a talk about history of Linux Containers during
<a href="https://devopsdays.pl/">Devopsdays Poland</a>.</p>
<p>Preparing this presentation was very interesting for me - I had to go through
quite a history of computing, understand business cases behind some
decisions and explain that to the audience. In the end it was a bit chaotic,
however feedback was great.</p>
<p>You can watch video here:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/_GSLj-c_LMI" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>And slides are here:</p>
<p><a href="https://maciej.lasyk.info/slides/2018-devopsdays-warsaw-quo-vadis-containers/">https://maciej.lasyk.info/slides/2018-devopsdays-warsaw-quo-vadis-containers/</a></p>
<p>Enjoy!</p>MariaDB multi master replication2019-02-14T00:00:00+01:002019-02-14T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2019-02-14:2019/Feb/14/mariadb-multi-master-replication/<p><center><img alt="MariaDB" src="https://maciej.lasyk.info/images/mariadb-logo.png" /></center></p>
<h2>What this is about</h2>
<p>So I have couple of MariaDB servers and wanted to take backups of some of
databases hosted on those servers. I could simply schedule an <a href="https://www.freedesktop.org/software/systemd/man/systemd.timer.html">systemd-timer</a>
job on every of those machine that would simply <a href="https://mariadb.com/kb/en/library/mysqldump/">mysqldump</a>
required databases on regular basis - let's say every our. And then send it
somewhere. Or make external service fetch those backups regularly.</p>
<p>But I don't like this idea. First of all because it increases my <a href="https://www.druva.com/blog/understanding-rpo-and-rto/">RPO</a>,
and secondly - in case one of those DB servers is down I'd have to redeploy it,
load backups etc. It takes time.</p>
<p>So instead of this idea (an old one, mentioned years ago in 2010 in <a href="http://shop.oreilly.com/product/9780596807290.do">Mysql High Availability</a>)
is to replicate all databases to a slave servers and take backups of all slave
DBs.</p>
<p>However this wasn't easy to perform back then in MySQL, because there was no
such thing as multi - master replication. Basically you can't start replication
from more than one master. In MySQL you had to run separate slave server for
each master you replicated.</p>
<p>And in MariaDB 10 You can finally do this. This image show the idea (I took it
from mariadb.com page):</p>
<p><center><img alt="MariaDB" src="https://maciej.lasyk.info/images/mariadb-multi_source_replication_small.png" /></center></p>
<h2>Ok, how to do this?</h2>
<p>In my scenario I had 2 already running MariaDB 10.x servers (from Fedora
upstream) and a fresh one, that I pointed to be the slave for above masters.</p>
<p>Also a downtime for each master is needed for a time of taking full backup. I
didn't find an easy way to get around this. If you do - please leave a
comment - thx!</p>
<p>Last, but not least - I used mariadb-server version <strong>10.1.26</strong>.</p>
<h4>1. Take backups of both masters.</h4>
<p>Should I really write this point? ;)</p>
<h4>2. Prepare your masters' configurations</h4>
<p>Now we need to make both master servers actual masters :) In order to do that
I edited <strong>/etc/my.cnf.d/mariadb-server.cnf</strong> (find your own <strong>my.cnf</strong> file -
probably in <strong>/etc/my.cnf</strong>) and put there something like this (of course leave
other configuration options that lay there):</p>
<div class="highlight"><pre><span></span><span class="c1"># first master server my.cnf file</span>
<span class="o">[</span>server<span class="o">]</span>
log-bin
<span class="nv">server_id</span><span class="o">=</span><span class="m">1</span>
log-basename<span class="o">=</span>master_rss
binlog-ignore-db<span class="o">=</span>mysql
<span class="o">[</span>mysqld<span class="o">]</span>
gtid-domain-id<span class="o">=</span><span class="m">1</span>
gtid-ignore-duplicates<span class="o">=</span>ON
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># second master server my.cnf file</span>
<span class="o">[</span>server<span class="o">]</span>
log-bin
<span class="nv">server_id</span><span class="o">=</span><span class="m">2</span>
log-basename<span class="o">=</span>master_retromtb
binlog-ignore-db<span class="o">=</span>mysql
<span class="o">[</span>mysqld<span class="o">]</span>
gtid-domain-id<span class="o">=</span><span class="m">2</span>
gtid-ignore-duplicates<span class="o">=</span>ON
</pre></div>
<p>So a bit of explanation:</p>
<ul>
<li><strong>log-bin</strong> - thanks to this server will become master and start logging
binary log. This means, that a log of changes will be saved to files and later
sent to slaves and replayed there. See <a href="https://mariadb.com/kb/en/library/activating-the-binary-log/">this article</a>
for more information.</li>
<li><strong>server_id</strong> - basically keep this value unique for every server in your
replication topology. This must be a number.</li>
<li><strong>log-basename</strong> - will be used as a part of name for log-bin files</li>
<li><strong>binlog-ignore-db</strong> - this tells which databases must not be replicated.
Very useful when you want to replicate all DBs but for some set of system DBs.</li>
<li><strong>gtid-domain-id</strong> - basically keep this unique for every master. <a href="https://mariadb.com/kb/en/library/gtid/#use-with-multi-source-replication-and-other-multi-master-setups">Here you
can read</a>
details about GTIDs in multi - source replication topologies.</li>
<li><strong>gtid-ignore-duplicates</strong> - this is actually not very needed in my
replication topology. Thanks to this setting when slave receives event with
GTID that was already processed it will ignore it. It is possible in situation
when you have a chain of master-slave/master-slave replications.</li>
</ul>
<h4>3. Prepare masters data for sending to slave</h4>
<p>Execute this for every master. I did it on both of my servers:</p>
<ol>
<li>This procedure execution time mostly depends on how long it takes to take
full backup of required databases. You can test it by running <strong>time mysqldump --databases db1,db2,db3 -u some_user -p > dbs.sql</strong>
beforehand.</li>
<li>Restart Mariadb service in order to apply above configurations (this might
be not needed as probably you can set above setting from mariadb SQL console
during runtime - you can easily find how to do that in the internets; just a
note for people who wanna minimize downtime).</li>
<li>Log into Mariadb SQL console and execute: <strong>FLUSH TABLES WITH READ LOCK;</strong> -
this will lock and disable all writes to this server. Prior to doing that I
always shutdown HTTP services or put them in maintenance mode. Keep in mind
that also some cron/timers services might try to connect. Thus locking on DB
layer is most save.</li>
<li>Execute <strong>select @@gtid_binlog_pos;</strong> and save this position for later use.</li>
<li>Take backup of required databases: <strong>mysqldump --databases db1,db2,db3 -u some_user -p > dbs.sql</strong></li>
<li>When this is finished unlock tables in SQL console: <strong>UNLOCK TABLES;</strong></li>
<li>Copy <strong>dbs.sql</strong> to the slave server.</li>
<li>Create replica user: <strong>GRANT REPLICATION SLAVE ON <em>.</em> TO 'replica_user'@'%' IDENTIFIED BY 'replica_password';</strong></li>
</ol>
<h4>4. Prepare slave server configuration</h4>
<p>My <strong>/etc/my.cnf.d/mariadb-server.cnf</strong> looks like this (removed everything
that was not connected to replication):</p>
<div class="highlight"><pre><span></span><span class="o">[</span>server<span class="o">]</span>
<span class="nv">server_id</span><span class="o">=</span><span class="m">1000</span>
<span class="o">[</span>mysqld<span class="o">]</span>
gtid-ignore-duplicates<span class="o">=</span>ON
</pre></div>
<p>So as you can see - a very simple config.</p>
<p>Now you need to upload both backups:</p>
<div class="highlight"><pre><span></span>mysql<span class="w"> </span>-u<span class="w"> </span>some_user<span class="w"> </span>-p<span class="w"> </span><<span class="w"> </span>dbs.sql
mysql<span class="w"> </span>-u<span class="w"> </span>some_user<span class="w"> </span>-p<span class="w"> </span><<span class="w"> </span>dbs2.sql
</pre></div>
<p>Having it uploaded it's time for starting replication:</p>
<ol>
<li>Execute <strong>SHOW ALL SLAVES STATUS\G</strong> - this should return an empty set (no
replication running for now).</li>
<li>Restart Mariadb service so new config is applied.</li>
<li>Run <strong>SET GLOBAL gtid_slave_pos = "1-1-X,2-2-Y";</strong> replacing each position
with proper you saved during taking backup on master.</li>
<li>For each master server setup replication: CHANGE MASTER 'master_name'
TO master_host="master_ip_addr", master_port=3306, master_user="replica_user",
master_use_gtid=current_pos, master_password='replica_user_pwd';</li>
<li>Now in order to start replication simply run: <strong>START ALL SLAVES;</strong> and
see if no problems were reported: <strong>SHOW WARNINGS;</strong></li>
<li>See replica status: <strong>SHOW ALL SLAVE STATUS\G;</strong></li>
</ol>
<p>And that's it.</p>
<h4>Debugging and solving problems</h4>
<p>You will probably hit some problems during setting this up for the first time.
Some helpful commands:</p>
<h5>Resetting master</h5>
<p>E.g. in order to create once again data backups for uploading to slaves: </p>
<p><strong>RESET MASTER</strong> (run this on master)</p>
<h5>Resetting one slave</h5>
<p>E.g. in order to recreate and reconfigure particular replication:</p>
<ol>
<li><strong>RESET SLAVE 'master_name' all</strong></li>
<li>afterwards you need to <strong>STOP ALL SLAVES;</strong> </li>
<li>set proper <strong>gtid_slave_pos</strong> by **SET GLOBAL... (check current GTID and replace the incorrect part with the proper one)</li>
<li>again <strong>CHANGE MASTER...</strong> and <strong>START SLAVE master_name;</strong></li>
</ol>
<h5>If you encounter replication error</h5>
<p>... and resolve it somehow than in order to skip it you will need to:</p>
<ol>
<li>**STOP SLAVE 'master_name'; </li>
<li>SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; </li>
<li>START SLAVE 'master_name';**</li>
</ol>
<p>This way you can skip one error (increase error counter to skip more than one)</p>
<h2>Helpful resources</h2>
<p>You can read about it more here:</p>
<ul>
<li><a href="https://mariadb.com/kb/en/library/multi-source-replication/">https://mariadb.com/kb/en/library/multi-source-replication/</a></li>
<li><a href="https://mariadb.com/kb/en/library/multi-master-replication/">https://mariadb.com/kb/en/library/multi-master-replication/</a></li>
<li><a href="https://mariadb.com/kb/en/library/gtid/">https://mariadb.com/kb/en/library/gtid/</a></li>
<li><a href="https://mariadb.com/resources/blog/mariadb-10-gtid-explained/">https://mariadb.com/resources/blog/mariadb-10-gtid-explained/</a></li>
<li><a href="https://www.percona.com/community-blog/2018/09/10/multi-master-with-mariadb-10-tutorial/">https://www.percona.com/community-blog/2018/09/10/multi-master-with-mariadb-10-tutorial/</a></li>
</ul>pip: bad interpreter: Permission denied2018-12-12T00:00:00+01:002018-12-12T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2018-12-12:2018/Dec/12/pip-bad-interpreter-permission-denied/<p>So lately while automating some jobs with <a href="https://www.rundeck.org">Rundeck</a>
I came across a problem:</p>
<div class="highlight"><pre><span></span>/tmp/11637-10898-rundeck.***-dispatch-script.tmp.sh:
/var/lib/rundeck/workspaces/f86853fb-a1e1-4517-bad8-a17931726d00/10898/
some-directory/ansible-playbooks/some-other-directory/plays/
virtual-env-directory/bin/pip:
bad<span class="w"> </span>interpreter:<span class="w"> </span>Permission<span class="w"> </span>denied
</pre></div>
<p>Wow, where did it come from?</p>
<p>This pip command works perfectly from the CLI. So why under
Rundeck it fails?</p>
<p>So I found out, that this is due to shebang length limitation. On Linux it just
can't be longer than 128 characters (<a href="https://github.com/torvalds/linux/blob/master/include/uapi/linux/binfmts.h#L19">see BINPRM_BUF_SIZE
here</a>).</p>
<p>And Rundeck changes this path a bit (this is due to job configuration I use).</p>
<p>So as I didn't want to recompile my Kernel for that to make it work I simply
made sure that the whole shebang would fit the 128 chars limit by shortening
the path mentioned in error above.</p>
<p>There's also another way - one could create a wrapper script that would run
the original one and run it via <strong>exec</strong> command.</p>
<p>You can read about it more here:</p>
<ul>
<li><a href="https://github.com/pypa/virtualenv/issues/596">https://github.com/pypa/virtualenv/issues/596</a></li>
<li><a href="https://stackoverflow.com/questions/10813538/shebang-line-limit-in-bash-and-linux-kernel">https://stackoverflow.com/questions/10813538/shebang-line-limit-in-bash-and-linux-kernel</a></li>
<li><a href="https://www.in-ulm.de/~mascheck/various/shebang/">https://www.in-ulm.de/~mascheck/various/shebang/</a></li>
</ul>Override DNS records with BIND and Response Policy Zones2018-12-08T00:00:00+01:002018-12-08T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2018-12-08:2018/Dec/08/override-dns-records-with-bind-and-response-policy-zones/<h2>What this is about</h2>
<p>I'm hosting my own BIND server on my local network. It's a caching - resolving
server, that speeds up my whole home network.</p>
<p>It works across couple of VLANs and provides different <a href="https://kb.isc.org/docs/aa-00851">views of DNS zones</a>.</p>
<p>However I have couple of public domains (e.g. lasyk.info) which are hosted in
public DNS providers (in this case it's <a href="https://www.cloudflare.com0">Cloudflare</a>).
And I can't use BIND zones there.</p>
<p>So now I have a <strong>service.lasyk.info</strong> hosted on my local network but the
A record for this service setup in Cloudflare is my public IP address of my
home network. And I can't access it from within this network because I don't
have such complicated routing setup (and I don't want to). Basically I'd need
to route my requests to this local service outside of my network and then back
inside. Quite stupid.</p>
<p>So I want to do some magic with my local BIND and make it override some of
public DNS entries. E.g. this <strong>service.lasyk.info</strong> should return A record
pointing at local IP address.</p>
<p>And this is exactly what Response Policy Zones in BIND are all about. All over
since 2010 :) </p>
<h2>So, how?</h2>
<p>It's simple. In you BIND server (at least 9.8.0 version) simply define new
zone:</p>
<div class="highlight"><pre><span></span>/etc/named.conf
---
zone<span class="w"> </span><span class="s2">"rpz"</span><span class="w"> </span><span class="o">{</span><span class="w"> </span>
<span class="w"> </span><span class="nb">type</span><span class="w"> </span>master<span class="p">;</span><span class="w"> </span>
<span class="w"> </span>file<span class="w"> </span><span class="s2">"/etc/named/overrides.rpz"</span><span class="p">;</span><span class="w"> </span>
<span class="w"> </span>notify<span class="w"> </span>yes<span class="p">;</span><span class="w"> </span>
<span class="c1"># also-notify { ip.of.your.secondaRy.DNS; };</span>
<span class="c1"># allow-transfer { ip.of.your.secondary.DNS; };</span>
<span class="o">}</span><span class="p">;</span>
<span class="c1"># and in options section</span>
options<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="c1"># your other options go here</span>
<span class="w"> </span><span class="c1"># ...</span>
<span class="w"> </span><span class="c1"># ...</span>
<span class="w"> </span><span class="c1"># overrides zone: </span>
<span class="w"> </span>response-policy<span class="w"> </span><span class="o">{</span><span class="w"> </span>zone<span class="w"> </span><span class="s2">"rpz"</span><span class="p">;</span><span class="w"> </span><span class="o">}</span><span class="p">;</span>
<span class="o">}</span>
</pre></div>
<p>and define this zone file:</p>
<div class="highlight"><pre><span></span>/etc/named/overrides.rpz
---
<span class="nv">$TTL</span><span class="w"> </span><span class="m">600</span>
@<span class="w"> </span>IN<span class="w"> </span>SOA<span class="w"> </span>localhost.<span class="w"> </span>root.localhost.<span class="w"> </span><span class="o">(</span>
<span class="w"> </span><span class="m">2018120300</span><span class="w"> </span><span class="p">;</span><span class="w"> </span>serial
<span class="w"> </span>1h<span class="w"> </span><span class="p">;</span><span class="w"> </span>refresh
<span class="w"> </span>30m<span class="w"> </span><span class="p">;</span><span class="w"> </span>retry
<span class="w"> </span>1w<span class="w"> </span><span class="p">;</span><span class="w"> </span>expiry
<span class="w"> </span>30m<span class="o">)</span><span class="w"> </span><span class="p">;</span><span class="w"> </span>minimum
<span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>desi.ns.cloudflare.com.
<span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>sean.ns.cloudflare.com.
service.lasyk.info<span class="w"> </span>A<span class="w"> </span><span class="m">192</span>.168.1.20
service2.lasyk.info<span class="w"> </span>CNAME<span class="w"> </span>google.com
</pre></div>
<p>Now after restarting BIND server you can dig it to make sure it's overriding
service.lasyk.info name with IP A record defined above.</p>
<h2>Extras</h2>
<p>You can also define custom logging channel for this RPZ zone in your BIND
config:</p>
<div class="highlight"><pre><span></span>/etc/named.conf
---
<span class="w"> </span>channel<span class="w"> </span>rpz-queries<span class="w"> </span><span class="o">{</span>
<span class="w"> </span>file<span class="w"> </span><span class="s2">"/var/log/named/queries-rpz.log"</span><span class="w"> </span>versions<span class="w"> </span><span class="m">10</span><span class="w"> </span>size<span class="w"> </span>500k<span class="p">;</span>
<span class="w"> </span>severity<span class="w"> </span>dynamic<span class="p">;</span><span class="w"> </span>
<span class="w"> </span>print-time<span class="w"> </span>yes<span class="p">;</span><span class="w"> </span>
<span class="w"> </span><span class="o">}</span><span class="p">;</span>
<span class="w"> </span>category<span class="w"> </span>rpz<span class="w"> </span><span class="o">{</span><span class="w"> </span>rpz-queries<span class="p">;</span><span class="w"> </span><span class="o">}</span><span class="p">;</span>
</pre></div>
<h2>Summary</h2>
<p>That's it. Very simple and effective. I didn't know about this feature, as
I learnt most of my DNS skills from <a href="http://shop.oreilly.com/product/9780596100575.do">DNS and BIND, 5th edition</a>
that was released in 2009 and covered BIND 9.3.2 and above feature comes with
9.8.0</p>Minecraft, Python and Fedora2018-02-04T00:00:00+01:002018-02-04T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2018-02-04:2018/Feb/04/minecraft-python-and-fedora/<p><center><img alt="Minecraft and Python" src="https://maciej.lasyk.info/images/minecraft-python-book.jpg" /></center></p>
<h2>What this is about</h2>
<p>Lately I bought above book as I wanted to start teaching my 9 - years old kid
how to script a bit w/Python in his beloved Minecraft world.</p>
<p>The book is great, but for one thing. Its first part is about installing
Minecraft, Spigot server and Python API on Windows or Mac or Raspberry-PI
(w/some Debian/Ubuntu based distro on board).</p>
<p>As we run Fedoras on our laptops (and on my server that is a physical machine
placed in our basement) I had to somehow make it work in our environment.</p>
<h2>Scenario</h2>
<p>So the scenario here is that Minecraft client (game) will be run on laptops
while Spigot server will be run on the Fedora 27 server inside of
systemd-nspawn container. Thanks to running it on container we will not
contaminate server's libraries. </p>
<p>You may of course install all of this on your laptop instead having server
and game running on same host (localhost).</p>
<h2>Creating systemd-nspawn container for Minecraft server</h2>
<p>This step is not needed at all. I just wanted to have Minecraft server
installed under systemd-nspawn container. I will not elaborate here about
creating containers as this is totally another matter.</p>
<p>If you'd like to also create systemd-nspawn container you may find following
files helpful:</p>
<ul>
<li><a href="https://github.com/docent-net/minecraft/blob/master/server/mkosi.default">mkosi.default</a>
for building container image</li>
<li><a href="https://github.com/docent-net/minecraft/blob/master/server/minecraft.nspawn">minecraft.nspawn</a>
for creating nspawn container</li>
<li><a href="https://github.com/docent-net/minecraft/blob/master/server/minecraft-server.service">minecraft-server.service</a>
use it as unit file for your service</li>
</ul>
<h2>Installing Spigot server on Fedora</h2>
<p>This one is actually simple. First install all prerequisites:</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>dnf<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>java-9-openjdk<span class="w"> </span>java-9-openjdk-devel<span class="w"> </span>git<span class="w"> </span>maven<span class="w"> </span>
</pre></div>
<p>Now download server libraries (preferably as user <strong>minecraft</strong> to
/home/minecraft/minecraft):</p>
<div class="highlight"><pre><span></span>mkdir<span class="w"> </span>/home/minecraft/minecraft
<span class="nb">cd</span><span class="w"> </span>/home/minecraft/minecraft
wget<span class="w"> </span><span class="s2">"https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar"</span>
</pre></div>
<p>And now build it - <strong>provide same version as you have on your Minecraft game</strong>:</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>config<span class="w"> </span>--global<span class="w"> </span>--unset<span class="w"> </span>core.autocrlf
java<span class="w"> </span>-jar<span class="w"> </span>BuildTools.jar<span class="w"> </span>--rev<span class="w"> </span><span class="m">1</span>.12.2
</pre></div>
<p>After a while Your server is all built up. Now you need to install
RaspberryJuice plugin for Bukkit that actually create socket API that will be
used by Python API helper:</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/zhuowei/RaspberryJuice<span class="w"> </span>/home/minecraft/minecraft/RaspberryJuice
<span class="nb">cd</span><span class="w"> </span>/home/minecraft/minecraft/RaspberryJuice
mvn<span class="w"> </span>package
</pre></div>
<p>When above is ready you need to copy freshly built jar to Spigot plugins
directory (use latest version of the built file; mine was 1.11):</p>
<div class="highlight"><pre><span></span>mkdir<span class="w"> </span>-p<span class="w"> </span>/home/minecraft/minecraft/plugins
cp<span class="w"> </span>/home/minecraft/minecraft/RaspberryJuice/jars/raspberryjuice-1.11.jar<span class="w"> </span>/home/minecraft/minecraft/plugins/<span class="w"> </span>
</pre></div>
<p>Now you need to confirm that you agree to all terms (haha lol) by creating
eula.txt file:</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span><span class="s2">"eula=true"</span><span class="w"> </span>><span class="w"> </span>/home/minecraft/minecraft/eula.txt
</pre></div>
<p>Afterwards you just need to start the server. You may create <strong>start.sh</strong>
script with following contents (remember - provide here same version as your
Spigot server and Minecraft client):</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
java<span class="w"> </span>-Xms512M<span class="w"> </span>-Xmx1024M<span class="w"> </span>-jar<span class="w"> </span>spigot-1.12.2.jar
</pre></div>
<p>or use system unit file <a href="(https://github.com/docent-net/minecraft/blob/master/server/minecraft-server.service)">I provided on Github</a></p>
<h2>Installing Python API on Fedora</h2>
<p>Now we need to install Python API library. Most internet guides tells about
running <strong>sudo pip3 install ...</strong>. I don't like it as this would pollute your
global Python path with some Minecraft API library. C'mon.</p>
<p>So imo we should rather create virtualenv for this:</p>
<div class="highlight"><pre><span></span>mkvirtualenv<span class="w"> </span>--python<span class="o">=</span>/usr/bin/python3<span class="w"> </span>minecraft
</pre></div>
<p>And now we can safely install the API library:</p>
<div class="highlight"><pre><span></span>workon<span class="w"> </span>minecraft
wget<span class="w"> </span>https://github.com/py3minepi/py3minepi/archive/master.zip
unzip<span class="w"> </span>master.zip
pip3<span class="w"> </span>install<span class="w"> </span>./py3minepi-master
</pre></div>
<p>And that's all. Now the following code run from your virtualenv should just
work:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">mcpi.minecraft</span> <span class="kn">import</span> <span class="n">Minecraft</span>
<span class="c1"># Provide proper IP of the server when not on localhost:</span>
<span class="c1"># mc = Minecraft.create('192.168.8.21')</span>
<span class="n">mc</span> <span class="o">=</span> <span class="n">Minecraft</span><span class="o">.</span><span class="n">create</span><span class="p">()</span>
<span class="n">mc</span><span class="o">.</span><span class="n">postToChat</span><span class="p">(</span><span class="s1">'Hello server!'</span><span class="p">)</span>
</pre></div>Devops meetups @Kraków2018-01-17T00:00:00+01:002018-01-17T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2018-01-17:2018/Jan/17/devops-meetups-krakow/<p><center><img alt="PID1 for the win!" src="https://maciej.lasyk.info/images/pid1.png" /></center></p>
<h2>Meetups</h2>
<p>So I'll be giving 2 talks today:</p>
<ul>
<li><a href="https://www.meetup.com/DevOpsKRK/events/246479142/">Google Cloud Platform and Terraform: diary of a combatant</a></li>
<li>story about creating stateless HA service using GCP services and
provisioning w/Terraform</li>
<li><a href="https://www.meetup.com/SysOpsPolska/events/245911108/?eventId=245911108">More about systemd</a></li>
</ul>
<p>URLs for slides:</p>
<ul>
<li><a href="https://maciej.lasyk.info/slides/devopskrk-gcp-terraform/">Terraform and GCP</a></li>
<li><a href="https://maciej.lasyk.info/slides/sysops-devops-systemd/">systemd</a></li>
</ul>Bokeh for Python data visualization2017-01-09T00:00:00+01:002017-01-09T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2017-01-09:2017/Jan/09/bokeh-for-python-data-visualization/<p><center><img alt="bokeh" src="https://maciej.lasyk.info/images/bokeh.png" /></center></p>
<p>Above image was generated by the following code:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">bokeh.plotting</span> <span class="kn">import</span> <span class="n">figure</span><span class="p">,</span> <span class="n">output_file</span><span class="p">,</span> <span class="n">show</span>
<span class="c1"># prepare some data</span>
<span class="n">N</span> <span class="o">=</span> <span class="mi">4000</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="n">N</span><span class="p">)</span> <span class="o">*</span> <span class="mi">100</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="n">N</span><span class="p">)</span> <span class="o">*</span> <span class="mi">100</span>
<span class="n">radii</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="n">N</span><span class="p">)</span> <span class="o">*</span> <span class="mf">1.5</span>
<span class="n">colors</span> <span class="o">=</span> <span class="p">[</span>
<span class="s2">"#</span><span class="si">%02x%02x%02x</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="n">r</span><span class="p">),</span> <span class="nb">int</span><span class="p">(</span><span class="n">g</span><span class="p">),</span> <span class="mi">150</span><span class="p">)</span> <span class="k">for</span> <span class="n">r</span><span class="p">,</span> <span class="n">g</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="mi">50</span><span class="o">+</span><span class="mi">2</span><span class="o">*</span><span class="n">x</span><span class="p">,</span> <span class="mi">30</span><span class="o">+</span><span class="mi">2</span><span class="o">*</span><span class="n">y</span><span class="p">)</span>
<span class="p">]</span>
<span class="c1"># output to static HTML file (with CDN resources)</span>
<span class="n">output_file</span><span class="p">(</span><span class="s2">"color_scatter.html"</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s2">"color_scatter.py example"</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s2">"cdn"</span><span class="p">)</span>
<span class="n">TOOLS</span><span class="o">=</span><span class="s2">"resize,crosshair,pan,wheel_zoom,box_zoom,reset,box_select,lasso_select"</span>
<span class="c1"># create a new plot with the tools above, and explicit ranges</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">figure</span><span class="p">(</span><span class="n">tools</span><span class="o">=</span><span class="n">TOOLS</span><span class="p">,</span> <span class="n">x_range</span><span class="o">=</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">100</span><span class="p">),</span> <span class="n">y_range</span><span class="o">=</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">100</span><span class="p">))</span>
<span class="c1"># add a circle renderer with vectorized colors and sizes</span>
<span class="n">p</span><span class="o">.</span><span class="n">circle</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">,</span> <span class="n">radius</span><span class="o">=</span><span class="n">radii</span><span class="p">,</span> <span class="n">fill_color</span><span class="o">=</span><span class="n">colors</span><span class="p">,</span> <span class="n">fill_alpha</span><span class="o">=</span><span class="mf">0.6</span><span class="p">,</span> <span class="n">line_color</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="c1"># show the results</span>
<span class="n">show</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
</pre></div>
<p>Thanks to <a href="http://bokeh.pydata.org/">Bokeh</a> library. Wow!</p>
<p>Above example was taken from <a href="http://bokeh.pydata.org/en/latest/docs/user_guide/quickstart.html#userguide-quickstart">Bokeh quickstart</a></p>
<p>Yup - I got new friend. Plotting with Bokeh looks very interesting in terms of
visualizing huge, browsable dataset.</p>systemd 231 in Centos 7 thx to Facebook2016-12-16T00:00:00+01:002016-12-16T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-12-16:2016/Dec/16/systemd-231-latest-in-centos-7-thx-to-facebook/<p><center><img alt="systemd ftw!" src="https://maciej.lasyk.info/images/pid1.png" /></center></p>
<h1>What is it about?</h1>
<p>So <a href="https://www.centos.org/">Centos7</a> currently has <a href="https://www.freedesktop.org/wiki/Software/systemd/">systemd</a>
version 219 installed which was released on 2015-02-16 (see <a href="https://github.com/systemd/systemd/blob/master/NEWS">NEWS</a>).</p>
<p>This is a huge problem, as we miss a lot of very important functions related to
journald, networkd, machinectl, systemd-nspawn and so on.</p>
<p>Porting latest systemd versions to Centos is a daunting task. It's possible,
but it takes time. And during latest <a href="https://conf.systemd.io/">systemd conference</a>
guys from <a href="https://twitter.com/fb_engineering">Facebook Engineering</a> told us,
that they actually did it. <a href="https://twitter.com/marcinskarbek">Marcin</a> asked
them if they're gonna share it and Davide replied, that it should be easy and
will think about it.</p>
<p>And they did. Yesterday.</p>
<p>On Facebook Incubator project on Github you'll find <a href="https://github.com/facebookincubator/rpm-backports">rpm - backports</a></p>
<p>And there's also a <a href="https://copr.fedorainfracloud.org/coprs/jsynacek/systemd-backports-for-centos-7/">COPR repo</a>
shared by a Red-Hatter <a href="https://github.com/jsynacek">Jan Synacek</a>.</p>
<p>Having those both you may install systemd 231 on your Centos7.</p>
<p>It's not currently meant to be running as production and also doesn't
provide SELinux policies / contexts. But.. it works. And it works on FB scale
:)</p>
<h1>Quick howto</h1>
<p>(it's copy&paste from <a href="https://copr.fedorainfracloud.org/coprs/jsynacek/systemd-backports-for-centos-7/">Jan's COPR installation instructions</a>
, so make sure it's up to date:</p>
<div class="highlight"><pre><span></span>Make<span class="w"> </span>sure<span class="w"> </span>to<span class="w"> </span>edit<span class="w"> </span>/etc/selinux/config<span class="w"> </span>and<span class="w"> </span>put<span class="w"> </span>SELinux<span class="w"> </span>to<span class="w"> </span>permissive<span class="w"> </span>before<span class="w"> </span>you<span class="w"> </span>update,<span class="w"> </span>otherwise<span class="w"> </span>your<span class="w"> </span>system<span class="w"> </span>will<span class="w"> </span>not<span class="w"> </span>boot<span class="w"> </span>anymore!
<span class="c1"># wget https://copr.fedorainfracloud.org/coprs/jsynacek/systemd-backports-for-centos-7/repo/epel-7/jsynacek-systemd-backports-for-centos-7-epel-7.repo -O /etc/yum.repos.d/jsynacek-systemd-centos-7.repo</span>
<span class="c1"># yum update systemd</span>
</pre></div>
<h1>Short story longer</h1>
<p>And special thanks to <a href="https://twitter.com/odcinek">Marcin Sawicki</a> for
catching my tweet and moving things forward <3</p>
<p>Guys - thank you a lot for this!</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Wow:<a href="https://t.co/Nh1yoxlGIG">https://t.co/Nh1yoxlGIG</a> now just waiting till <a href="https://twitter.com/fb_engineering">@fb_engineering</a> share theirs <a href="https://twitter.com/hashtag/systemd?src=hash">#systemd</a> <a href="https://twitter.com/hashtag/Fedora?src=hash">#Fedora</a> port to <a href="https://twitter.com/hashtag/centos?src=hash">#centos</a> mentioned on <a href="https://twitter.com/systemdconf">@systemdconf</a> <3</p>— Maciek Lasyk (@docent_net) <a href="https://twitter.com/docent_net/status/806196636770795521">December 6, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/docent_net">@docent_net</a> was it Davide announcing that? I can ping him tmorrow at the office ;)</p>— Marcin Sawicki (@odcinek) <a href="https://twitter.com/odcinek/status/809314228091785216">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/odcinek">@odcinek</a> <a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> do you remember? If not I'll search it in the videos.</p>— Maciek Lasyk (@docent_net) <a href="https://twitter.com/docent_net/status/809332228723445760">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/docent_net">@docent_net</a> <a href="https://twitter.com/odcinek">@odcinek</a> no, not announced. It was only an answer about possibility of releasing when we asked about this.</p>— Marcin Skarbek (@marcinskarbek) <a href="https://twitter.com/marcinskarbek/status/809334484533387264">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> <a href="https://twitter.com/odcinek">@odcinek</a> 'mentioned' -> 'announced' -> 'published'; can we agree on that? :D</p>— Maciek Lasyk (@docent_net) <a href="https://twitter.com/docent_net/status/809350242223198208">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="und" dir="ltr"><a href="https://twitter.com/docent_net">@docent_net</a> <a href="https://twitter.com/odcinek">@odcinek</a> <a href="https://t.co/7q1ZAXnSj7">pic.twitter.com/7q1ZAXnSj7</a></p>— Marcin Skarbek (@marcinskarbek) <a href="https://twitter.com/marcinskarbek/status/809351156375883776">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> <a href="https://twitter.com/odcinek">@odcinek</a> This is part of presentation/Q&A where systemd port was mentioned: <a href="https://t.co/NziIK41PYE">https://t.co/NziIK41PYE</a> but have to dig where</p>— Maciek Lasyk (@docent_net) <a href="https://twitter.com/docent_net/status/809352168725684226">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> <a href="https://twitter.com/odcinek">@odcinek</a> Davide actually told that FB will share it. My memory tells me it was somewhere inside the presentation, not Q&A</p>— Maciek Lasyk (@docent_net) <a href="https://twitter.com/docent_net/status/809352439266758656">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/docent_net">@docent_net</a> <a href="https://twitter.com/odcinek">@odcinek</a> ok, I found it: <a href="https://t.co/gMMSqbp8vk">https://t.co/gMMSqbp8vk</a></p>— Marcin Skarbek (@marcinskarbek) <a href="https://twitter.com/marcinskarbek/status/809357830201503744">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> <a href="https://twitter.com/odcinek">@odcinek</a> great, so now fingers crossed :)</p>— Maciek Lasyk (@docent_net) <a href="https://twitter.com/docent_net/status/809362164930805761">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="es" dir="ltr"><a href="https://twitter.com/docent_net">@docent_net</a> <a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> tada! <a href="https://t.co/yvMtY4zJSM">https://t.co/yvMtY4zJSM</a></p>— Marcin Sawicki (@odcinek) <a href="https://twitter.com/odcinek/status/809450352390991872">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/odcinek">@odcinek</a> <a href="https://twitter.com/docent_net">@docent_net</a> <a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> And the copr repo from RH of these packages for CentOS 7: <a href="https://t.co/F7EmQhxOZU">https://t.co/F7EmQhxOZU</a></p>— PhilD (@ThePhilD) <a href="https://twitter.com/ThePhilD/status/809492567511310336">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/ThePhilD">@ThePhilD</a> <a href="https://twitter.com/odcinek">@odcinek</a> <a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> sweet, many thanks! Next question was about <a href="https://twitter.com/hashtag/SELinux?src=hash">#SELinux</a>, but I see it's like work in progress ;) Thanks!</p>— Maciek Lasyk (@docent_net) <a href="https://twitter.com/docent_net/status/809494964958883840">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="und" dir="ltr"><a href="https://twitter.com/docent_net">@docent_net</a> <a href="https://twitter.com/ThePhilD">@ThePhilD</a> <a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> <a href="https://t.co/vLALC4IUZT">pic.twitter.com/vLALC4IUZT</a></p>— Marcin Sawicki (@odcinek) <a href="https://twitter.com/odcinek/status/809513208998309888">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/odcinek">@odcinek</a> <a href="https://twitter.com/ThePhilD">@ThePhilD</a> <a href="https://twitter.com/marcinskarbek">@marcinskarbek</a> booring :p</p>— Maciek Lasyk (@docent_net) <a href="https://twitter.com/docent_net/status/809515640193220608">December 15, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>Python + UnitTest + cProfile + Mock2016-12-14T00:00:00+01:002016-12-14T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-12-14:2016/Dec/14/python-unittest-cprofile-mock/<p><center><img alt="coding!" src="https://maciej.lasyk.info/images/python-profiling.png" /></center></p>
<h1>Problem?</h1>
<p>Another day in work and I find out that in one of our projects unit tests suite
executes in about <strong>13 seconds</strong>. Wow! It was like 200ms before, so what
happened?</p>
<p>Btw - this particular project is a Python
<a href="https://cloud.google.com/appengine/">Google/AppEngine</a> application. We use
<a href="https://docs.python.org/2/library/unittest.html">UnitTest</a> framework for unit
tests as well as <a href="https://pypi.python.org/pypi/mock/">mock</a> library.</p>
<p>One could say that it's not a problem as you don't test that often. Unless you
do. In <a href="http://www.ocadotechnology.com/">Ocado</a> we really do
<a href="https://en.wikipedia.org/wiki/Test-driven_development">TDD</a> everywhere. E.g.
in this particular project <strong>Guacamole</strong> we have <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks">git hooks</a>
that run unit - tests on every git commit. That's why we really want unit tests
to run very fast. <strong>Personally I think that unit test suite that executes in
more than 0,5s is broken</strong>.</p>
<p>One more thing. In order to run our tests we use <a href="https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/localtesting/runner.py">tests_runner.py</a>
provided by Google - it's a tool that loads GCP SDK as well as detects our
tests in project directory and run suite. It's important, as it's just another
complication.</p>
<p>So it looks like:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>python<span class="w"> </span>tests_runner.py<span class="w"> </span>--test-path<span class="w"> </span>tests/<span class="w"> </span>--test-pattern<span class="w"> </span><span class="s1">'test*.py'</span><span class="w"> </span>/home/somewhere/google-cloud-sdk
----------------------------------------------------------------------
Ran<span class="w"> </span><span class="m">47</span><span class="w"> </span>tests<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">13</span>.105s
OK
</pre></div>
<p>You see? <strong>13 seconds to run 47 tests - disgusting!</strong></p>
<h1>Tests look fine</h1>
<p>So I <strong>git log / diff</strong> and I saw that we've got couple of new tests. But each
of those tests brought something good to our projects, so it wasn't about
removing those tests, but rather checking where is the root cause.
<strong>I didn't want to remove any tests</strong> just because it executes slow. <strong>I
wanted to find out why</strong> tests run slow and fix it.</p>
<h1>Let's cProfile!</h1>
<p>So I thought that I'll try with profiling those tests. In Python there're
couple of ways to do it. I know great <a href="https://docs.python.org/2/library/profile.html">cProfile</a>
module:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>python<span class="w"> </span>-m<span class="w"> </span>cProfile<span class="w"> </span>-s<span class="w"> </span>totime<span class="w"> </span>tests_runner.py<span class="w"> </span>--test-path<span class="w"> </span>tests/<span class="w"> </span>--test-pattern<span class="w"> </span><span class="s1">'test*.py'</span><span class="w"> </span>/home/somewhere/google-cloud-sdk
----------------------------------------------------------------------
Ran<span class="w"> </span><span class="m">47</span><span class="w"> </span>tests<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">13</span>.304s
OK
<span class="w"> </span>ncalls<span class="w"> </span>tottime<span class="w"> </span>percall<span class="w"> </span>cumtime<span class="w"> </span>percall<span class="w"> </span>filename:lineno<span class="o">(</span><span class="k">function</span><span class="o">)</span>
<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="m">1</span>.445<span class="w"> </span><span class="m">0</span>.080<span class="w"> </span><span class="m">1</span>.445<span class="w"> </span><span class="m">0</span>.080<span class="w"> </span><span class="o">{</span>method<span class="w"> </span><span class="s1">'connect'</span><span class="w"> </span>of<span class="w"> </span><span class="s1">'_socket.socket'</span><span class="w"> </span>objects<span class="o">}</span>
<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="m">1</span>.310<span class="w"> </span><span class="m">0</span>.073<span class="w"> </span><span class="m">1</span>.310<span class="w"> </span><span class="m">0</span>.073<span class="w"> </span><span class="o">{</span>method<span class="w"> </span><span class="s1">'do_handshake'</span><span class="w"> </span>of<span class="w"> </span><span class="s1">'_ssl._SSLSocket'</span><span class="w"> </span>objects<span class="o">}</span>
<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="m">0</span>.833<span class="w"> </span><span class="m">0</span>.046<span class="w"> </span><span class="m">0</span>.833<span class="w"> </span><span class="m">0</span>.046<span class="w"> </span><span class="o">{</span>_socket.getaddrinfo<span class="o">}</span>
<span class="w"> </span><span class="m">9378</span><span class="w"> </span><span class="m">0</span>.757<span class="w"> </span><span class="m">0</span>.000<span class="w"> </span><span class="m">0</span>.757<span class="w"> </span><span class="m">0</span>.000<span class="w"> </span><span class="o">{</span>method<span class="w"> </span><span class="s1">'read'</span><span class="w"> </span>of<span class="w"> </span><span class="s1">'_ssl._SSLSocket'</span><span class="w"> </span>objects<span class="o">}</span>
<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="m">0</span>.186<span class="w"> </span><span class="m">0</span>.010<span class="w"> </span><span class="m">0</span>.186<span class="w"> </span><span class="m">0</span>.010<span class="w"> </span><span class="o">{</span>method<span class="w"> </span><span class="s1">'set_default_verify_paths'</span><span class="w"> </span>of<span class="w"> </span><span class="s1">'_ssl._SSLContext'</span><span class="w"> </span>objects<span class="o">}</span>
<span class="w"> </span><span class="m">36</span><span class="w"> </span><span class="m">0</span>.109<span class="w"> </span><span class="m">0</span>.003<span class="w"> </span><span class="m">0</span>.109<span class="w"> </span><span class="m">0</span>.003<span class="w"> </span>decoder.py:370<span class="o">(</span>raw_decode<span class="o">)</span>
<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="m">0</span>.096<span class="w"> </span><span class="m">0</span>.005<span class="w"> </span><span class="m">0</span>.096<span class="w"> </span><span class="m">0</span>.005<span class="w"> </span><span class="o">{</span>method<span class="w"> </span><span class="s1">'load_verify_locations'</span><span class="w"> </span>of<span class="w"> </span><span class="s1">'_ssl._SSLContext'</span><span class="w"> </span>objects<span class="o">}</span>
<span class="w"> </span><span class="m">305</span><span class="w"> </span><span class="m">0</span>.028<span class="w"> </span><span class="m">0</span>.000<span class="w"> </span><span class="m">0</span>.826<span class="w"> </span><span class="m">0</span>.003<span class="w"> </span>socket.py:410<span class="o">(</span>readline<span class="o">)</span>
...
<span class="o">(</span>cut<span class="w"> </span>rest<span class="w"> </span>as<span class="w"> </span>it<span class="err">'</span>s<span class="w"> </span>not<span class="w"> </span>that<span class="w"> </span>interesting<span class="o">)</span>
</pre></div>
<p>So as you can see - I sorted all calls by <strong>total time</strong>. So <strong>_socket.socket.connect()</strong>
was called <strong>18 times</strong> and it took circa <strong>1,5s</strong>. Wow! And another
<strong>2 seconds</strong> on SSL socket manipulations. Wow and WTF? I mean - unit tests
creating any socket connections and involving SSL processing?</p>
<p>But from this point I just see that some of libraries used in the whole stack
(in this situation _socket and _ssl) take a bunch of time to process some
request. But searching for those particular calls can be like searching for a
needle in the haystack. I could use <a href="https://docs.python.org/2/library/pdb.html">pdb</a>
debugger and debug those tests (or in Pycharm's builtin debugger that I find
much more handy) but still - debugging take a lot of time.</p>
<h1>Let's visualize problem!</h1>
<p>I love Python because it provides me with so many tools and libs created by
great community. So there is this <a href="https://github.com/jrfonseca/gprof2dot">gprof2dot</a>
converter that takes output from cProfile and generates SVG graphs visualizing
calls. Wow!</p>
<p>Actually this tool visualizes output from much more tools (see docs) - that
makes it very generic and valuable.</p>
<p>So I re-run profiler, wrote profile data to a file and generated an SVG file:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>python<span class="w"> </span>-m<span class="w"> </span>cProfile<span class="w"> </span>-o<span class="w"> </span>guacamole.pstats<span class="w"> </span>tests_runner.py<span class="w"> </span>--test-path<span class="w"> </span>tests/<span class="w"> </span>--test-pattern<span class="w"> </span><span class="s1">'test*.py'</span><span class="w"> </span>/home/somewhere/google-cloud-sdk
...
$<span class="w"> </span>gprof2dot<span class="w"> </span>-f<span class="w"> </span>pstats<span class="w"> </span>guacamole.pstats<span class="w"> </span><span class="p">|</span><span class="w"> </span>dot<span class="w"> </span>-Tsvg<span class="w"> </span>-o<span class="w"> </span>guacamole.svg
</pre></div>
<p>And I saw this (click to see larger SVG):</p>
<p><center><a href="https://maciej.lasyk.info/images/guacamole.svg"><img alt="Guacamole profiling" src="https://maciej.lasyk.info/images/guacamole-min.png" /></a>]</center></p>
<p>Wow! So now following coloured path we might see that:</p>
<p><center><img alt="Guacamole profiling" src="https://maciej.lasyk.info/images/guacamole-profiling.png" /></center></p>
<p>So actually <strong>test_Backup.py</strong> contains problematic tests (almost 87% of time
consumed). Moreover it is <strong>setUp</strong> fixture:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">test_Backup</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">testbed</span> <span class="o">=</span> <span class="n">testbed</span><span class="o">.</span><span class="n">Testbed</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">testbed</span><span class="o">.</span><span class="n">activate</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">Backup</span> <span class="o">=</span> <span class="n">Backup</span><span class="p">()</span>
</pre></div>
<p>And going further it is Backup constructor <strong>init</strong>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Backup</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">Storage</span> <span class="o">=</span> <span class="n">Storage</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">StorageTransfer</span> <span class="o">=</span> <span class="n">StorageTransfer</span><span class="p">()</span>
</pre></div>
<p>And even further it is about time taken to invoke constructors of <strong>Storage</strong>
and <strong>StorageTransfer</strong>. Let's see:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Storage</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">http</span> <span class="o">=</span> <span class="n">httplib2</span><span class="o">.</span><span class="n">Http</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service</span> <span class="o">=</span> <span class="n">googleapiclient</span><span class="o">.</span><span class="n">discovery</span><span class="o">.</span><span class="n">build</span><span class="p">(</span>
<span class="s1">'storage'</span><span class="p">,</span>
<span class="s1">'v1'</span><span class="p">,</span>
<span class="n">credentials</span><span class="o">=</span><span class="n">some_method</span><span class="p">(),</span>
<span class="n">http</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">http</span>
<span class="p">)</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">StorageTransfer</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">http</span> <span class="o">=</span> <span class="n">httplib2</span><span class="o">.</span><span class="n">Http</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service</span> <span class="o">=</span> <span class="n">googleapiclient</span><span class="o">.</span><span class="n">discovery</span><span class="o">.</span><span class="n">build</span><span class="p">(</span>
<span class="s1">'storagetransfer'</span><span class="p">,</span>
<span class="s1">'v1'</span><span class="p">,</span>
<span class="n">credentials</span><span class="o">=</span><span class="n">some_method</span><span class="p">(),</span>
<span class="n">http</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">http</span>
<span class="p">)</span>
</pre></div>
<p>So where's the problem? Above code is a <a href="https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/storage/transfer_service/create_client.py">standard authentication for GCP</a>
and we thought it will be mocked by <strong>testbed.activate()</strong> (provided by Google)
on some level. But as we can see - it wasn't. So every test we run there is an
instance of <strong>googleapiclient.discovery.build</strong> created - and this object
basically does all the authentication based requests (thus SSL, requests lib
etc involved).</p>
<h1>Solution - simple mock!</h1>
<p>So the solution was quite simple. We decided to mock whole
<strong>googleapiclient.discovery.build</strong> and that should do the trick:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">google.appengine.ext</span> <span class="kn">import</span> <span class="n">vendor</span>
<span class="kn">from</span> <span class="nn">mock</span> <span class="kn">import</span> <span class="n">Mock</span>
<span class="n">vendor</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s1">'lib'</span><span class="p">)</span>
<span class="kn">import</span> <span class="nn">googleapiclient.discovery</span> <span class="c1"># nopep8 pylint: disable=C0413</span>
<span class="k">class</span> <span class="nc">test_Backup</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">googleapiclient</span><span class="o">.</span><span class="n">discovery</span><span class="o">.</span><span class="n">build</span> <span class="o">=</span> <span class="n">Mock</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">testbed</span> <span class="o">=</span> <span class="n">testbed</span><span class="o">.</span><span class="n">Testbed</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">testbed</span><span class="o">.</span><span class="n">activate</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">Backup</span> <span class="o">=</span> <span class="n">Backup</span><span class="p">()</span>
</pre></div>
<p>And...:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>python<span class="w"> </span>tests_runner.py<span class="w"> </span>--test-path<span class="w"> </span>tests/<span class="w"> </span>--test-pattern<span class="w"> </span><span class="s1">'test*.py'</span><span class="w"> </span>/home/somewhere/google-cloud-sdk
----------------------------------------------------------------------
Ran<span class="w"> </span><span class="m">47</span><span class="w"> </span>tests<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.047s
OK
</pre></div>
<p>Yes! It worked like a charm! From almost 13 seconds down to 0.047 seconds.
<strong>That is a 27659% improvement, nice!</strong></p>
<h1>Foreword</h1>
<p>One more thing. Our original implementation of e.g. <strong>Storage()</strong> was:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">googleapiclient.discovery</span> <span class="kn">import</span> <span class="n">build</span>
<span class="k">class</span> <span class="nc">Storage</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">http</span> <span class="o">=</span> <span class="n">httplib2</span><span class="o">.</span><span class="n">Http</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service</span> <span class="o">=</span> <span class="n">build</span><span class="p">(</span>
<span class="s1">'storage'</span><span class="p">,</span>
<span class="s1">'v1'</span><span class="p">,</span>
<span class="n">credentials</span><span class="o">=</span><span class="n">some_method</span><span class="p">(),</span>
<span class="n">http</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">http</span>
<span class="p">)</span>
</pre></div>
<p>But we had to mock the whole loading context:</p>
<div class="highlight"><pre><span></span><span class="n">googleapiclient</span><span class="o">.</span><span class="n">discovery</span><span class="o">.</span><span class="n">build</span> <span class="o">=</span> <span class="n">Mock</span><span class="p">()</span>
</pre></div>
<p>This wouldn't work as it would mock <strong>build()</strong> from other namespace. So we
basically refactored a bit of <strong>Storage()</strong>:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">googleapiclient.discovery</span> <span class="c1"># nopep8 pylint: disable=C0413</span>
<span class="k">class</span> <span class="nc">Storage</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">http</span> <span class="o">=</span> <span class="n">httplib2</span><span class="o">.</span><span class="n">Http</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service</span> <span class="o">=</span> <span class="n">googleapiclient</span><span class="o">.</span><span class="n">discovery</span><span class="o">.</span><span class="n">build</span><span class="p">(</span>
<span class="s1">'storage'</span><span class="p">,</span>
<span class="s1">'v1'</span><span class="p">,</span>
<span class="n">credentials</span><span class="o">=</span><span class="n">some_method</span><span class="p">(),</span>
<span class="n">http</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">http</span>
<span class="p">)</span>
</pre></div>
<p>And now mocking started working. So - profile, visualize and mock! Keep your
unit - tests under 0,5s!</p>Coding katas on codewars2016-12-11T00:00:00+01:002016-12-11T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-12-11:2016/Dec/11/coding-katas-on-codewars/<p><center><img alt="coding!" src="https://maciej.lasyk.info/images/weekend-coding.jpg" /></center></p>
<p>I'm learning <a href="https://www.rust-lang.org">Rust</a> and reminding a bit of <strong>C</strong> as
I'm working on some side - projects with <strong>systemd</strong>. I always believed that
the only way to learn something is practise & repeat.</p>
<p>So I thought that I need some exercises. Found great service for that:
<a href="codewars.com">https://www,codewars.com</a>. You choose languages you want to
practise and start coding. By completing Katas you earn points, honor and some
other stuff. Gamification here - I like it!</p>
<p>Every Kata provides some tests you may use, so <a href="https://en.wikipedia.org/wiki/Test-driven_development">TDD</a>
is in place. You may also create your own Kata and publish it. </p>
<p>Try yourself. You may find me here: <a href="https://www.codewars.com/users/docent-net">https://www.codewars.com/users/docent-net</a></p>Wrocław devops meetup recap2016-11-30T00:00:00+01:002016-11-30T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-11-30:2016/Nov/30/wroclaw-devops-meetup-recap/<p><center><img alt="systemd devops meetup at Wrocław!" src="https://maciej.lasyk.info/images/wroclaw-devops-systemd.jpg" /></center></p>
<h1>How did it go?</h1>
<p>Well, I must admit that yet again it didn't go without unpredicted and stressful
moments. I run <a href="https://maciej.lasyk.info/2016/Nov/27/systemd-workshops-recap/">systemd workshops</a>
last weekend so I was rather fresh in terms of content knowledge. The only
thing left was preparing slides and making sure all demos work offline.</p>
<p>So I did. And about 2 hours before meetup beginning I realized, that demos are not
working because of some bad imports in Python code. It was a mess - even demos
from workshops didn't work. I asked my self WTF couple of times, but somehow
found, that this was because of <strong>dnf upgrade</strong> and some more global
<strong>pip installs</strong> related to systemd libs that I made 2 evenings before and even
forgot about it (yep, I work mainly on virtualenvs, but for some reason - not
this time). I'm working on a blog post explaining the winding road to
systemd/Python development (the hard part is about 4 different libraries being
there). I'll publish it within next few days.</p>
<p>So the feedback was really positive - thanks! Quite many people approached me
after presentation and told, that they learned much and would like to see more
(e.g. during workshops). That was really nice - feedback always appreciated :)</p>
<p>Btw - I told some of you about discussion on debian.org about systemd
integration. So <a href="https://wiki.debian.org/Debate/initsystem/systemd">here it is</a>
- very good reading. Srsly.</p>
<h1>What did I learn?</h1>
<ol>
<li>That even Debian / Ubuntu centric people are really keen to learn how
systemd might help them. E.g. <strong>KillMode</strong>, journal fields, boot process
analyzing, sd-notify Python bindings, dbus - communication & busctl
pcap/Wireshark export, handy nspawn containers and so many more.</li>
<li>That Python systemd libraries overlaps with each other. Not that big mess,
but I really have to sort it out and post about it and ask authors about
those overlappings.</li>
<li>That systemd hate is actually gone when you talk about details, listen to
people problems (and really guys - listen and understand) and explain
thoroughly. There's no hate when you treat others equally.</li>
<li>That reading couple of times <a href="http://0pointer.de/blog/projects/the-biggest-myths.html">The Biggest Myths</a>
about systemd by Lennart Poettering helped me much in terms of
talking to people and explaining them why systemd is created this way and
not other (thanks Lennart!)</li>
<li>That systemd conference in Berlin I attended 2 times in a row was a great
idea and that I will get there again and again. If you missed it - you
may <a href="https://www.youtube.com/channel/UCvq_RgZp3kljp9X8Io9Z1DA/videos">watch the videos</a></li>
</ol>
<h1>Slides & materials</h1>
<p>I published slides <a href="http://maciej.lasyk.info/slides/wroclaw_devops_systemd_2016/">here</a>
(not sure why it doesn't work on FF for me, try w/Chrome).</p>
<p>Codebase is on <a href="https://github.com/docent-net/systemd-workshop">systemd workshops repo</a></p>
<h1>Thank you guys!</h1>
<p>Thanks to organizers (Kasia Wojciechowska, Kamil Herbik & Tomek Tarczyński).
Great job with preparations - I love your underground - pub venue!</p>
<p>Thanks to Piotrek Kurpik from Ocado Technology for his great support. It's
always nice to have your guy on the side ;)</p>
<p>And many, many thanks to all attendees - for your presence, questions,
feedback and help during presentation when it was needed ;)</p>
<h2>Anything more? #learningsystemd!</h2>
<p>If you wanna learn more about <strong>systemd</strong> simply follow
<a href="/feeds/tag/learning-systemd.rss.xml">learning-systemd RSS</a>
or
<a href="/feeds/tag/learning-systemd.atom.xml">learning-systemd ATOM</a>
tag in this very blog (or on Twitter:
<a href="https://twitter.com/search?f=tweets&q=%23learningsystemd&src=typd">#learningsystemd</a>)</p>Enabling TAG feeds2016-11-29T00:00:00+01:002016-11-29T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-11-29:2016/Nov/29/enabling-tag-feeds/<p><center><img alt="RSS" src="https://maciej.lasyk.info/images/rss.png" /></center></p>
<h2>Wow, such feeds, very tags ^^</h2>
<p>So I decided to share some of my knowledge on cool and sometimes boring things
and start publishing posts that will show / teach you various subjects of
platform engineering.</p>
<p>In order to make it work I decided to provide RSS/ATOM feeds to specific tags
on this blog. The first one is <a href="/feeds/learning-systemd/">learning-systemd</a>.
Simply check the URL schema for <strong>learning-systemd</strong> tag feeds:</p>
<ul>
<li><a href="/feeds/tag/learning-systemd.rss.xml">learning-systemd RSS</a></li>
<li><a href="/feeds/tag/learning-systemd.atom.xml">learning-systemd ATOM</a></li>
</ul>
<h2>#learning-series</h2>
<p>As systemd is not only thing I wanna show you I've also created
<strong>learning-series</strong> tag, which will gather all <strong>learning-*</strong> blog posts:</p>
<ul>
<li><a href="/feeds/tag/learning-series.rss.xml">learning-series RSS</a></li>
<li><a href="/feeds/tag/learning-series.atom.xml">learning-series ATOM</a></li>
</ul>systemd mask2016-11-29T00:00:00+01:002016-11-29T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-11-29:2016/Nov/29/systemd-mask/<p><center><img alt="PID1 for the win!" src="https://maciej.lasyk.info/images/pid1.png" /></center></p>
<h2>Problem</h2>
<p>I come from Slackware and later RHEL / Fedora world. For me it's natural that
whenever I install any service it's not started until I say so.</p>
<p>So I was bit scarred, when long time ago on some Debian / Ubuntu I realized,
that after installing HTTPD server it started immediately. I didn't know until
it was registered within our HaProxy load - balancer starting servicing some
default content instead of it should be in case it was properly configured. </p>
<p>So it was really peculiar back then, that service may be started without any
confirmation even before it it properly configured!</p>
<h2>Looking for solution before systemd</h2>
<p>So I started digging and looking for an straightforward answer how to disable
"auto - starting upon installation". And I didn't find nothing really useful
beside some hacks.</p>
<p>However after digging even deeper I <a href="https://anonscm.debian.org/git/collab-maint/init-system-helpers.git/tree/script/deb-systemd-invoke#n70">found this</a>
- so looks like that starting from Ubuntu 16.04 there is a native method
implemented in <strong>policy-rc.d</strong>:</p>
<div class="highlight"><pre><span></span><span class="k">if</span><span class="w"> </span><span class="o">(</span>-x<span class="w"> </span><span class="nv">$policyhelper</span><span class="o">)</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span>my<span class="w"> </span><span class="nv">$unit</span><span class="w"> </span><span class="o">(</span>@units<span class="o">)</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>system<span class="o">(</span>qq<span class="p">|</span><span class="nv">$policyhelper</span><span class="w"> </span><span class="nv">$unit</span><span class="w"> </span><span class="s2">"</span><span class="nv">$action</span><span class="s2">"</span><span class="p">|</span><span class="o">)</span><span class="p">;</span>
<span class="w"> </span><span class="c1"># 0 or 104 means run</span>
<span class="w"> </span><span class="c1"># 101 means do not run</span>
<span class="w"> </span>my<span class="w"> </span><span class="nv">$exitcode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">(</span><span class="nv">$?</span><span class="w"> </span>>><span class="w"> </span><span class="m">8</span><span class="o">)</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">(</span><span class="nv">$exitcode</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">101</span><span class="o">)</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>print<span class="w"> </span>STDERR<span class="w"> </span><span class="s2">"</span><span class="nv">$policyhelper</span><span class="s2"> returned 101, not running '"</span><span class="w"> </span>.<span class="w"> </span>join<span class="o">(</span><span class="s1">' '</span>,<span class="w"> </span>@ARGV<span class="o">)</span><span class="w"> </span>.<span class="w"> </span><span class="s2">"'\n"</span><span class="p">;</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">0</span><span class="p">;</span>
<span class="w"> </span><span class="o">}</span><span class="w"> </span>elsif<span class="w"> </span><span class="o">(</span><span class="nv">$exitcode</span><span class="w"> </span>!<span class="o">=</span><span class="w"> </span><span class="m">104</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nv">$exitcode</span><span class="w"> </span>!<span class="o">=</span><span class="w"> </span><span class="m">0</span><span class="o">)</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>print<span class="w"> </span>STDERR<span class="w"> </span><span class="s2">"deb-systemd-invoke only supports </span><span class="nv">$policyhelper</span><span class="s2"> return codes 0, 101, and 104!\n"</span><span class="p">;</span>
<span class="w"> </span>print<span class="w"> </span>STDERR<span class="w"> </span><span class="s2">"Got return code </span><span class="nv">$exitcode</span><span class="s2">, ignoring.\n"</span><span class="p">;</span>
<span class="w"> </span><span class="o">}</span>
<span class="w"> </span><span class="o">}</span>
<span class="o">}</span>
</pre></div>
<p>So actually what's enough is:</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-e<span class="w"> </span><span class="s1">'#!/bin/bash\nexit 101'</span><span class="w"> </span>><span class="w"> </span>/usr/sbin/policy-rc.d
sudo<span class="w"> </span>chmod<span class="w"> </span><span class="m">755</span><span class="w"> </span>/usr/sbin/policy-rc.d
sudo<span class="w"> </span>/usr/sbin/policy-rc.d
</pre></div>
<p>After doing so <strong>no service will be started upon installation</strong>. So it's a
global change.</p>
<h2>how about systemd?</h2>
<p>So systemd provides <strong>mask</strong> method within <strong>systemctl</strong>. From the
documentation:</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="nv">mask</span><span class="w"> </span><span class="nv">NAME</span>...
<span class="w"> </span><span class="nv">Mask</span><span class="w"> </span><span class="nv">one</span><span class="w"> </span><span class="nv">or</span><span class="w"> </span><span class="nv">more</span><span class="w"> </span><span class="nv">units</span>,<span class="w"> </span><span class="nv">as</span><span class="w"> </span><span class="nv">specified</span><span class="w"> </span><span class="nv">on</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">command</span><span class="w"> </span><span class="nv">line</span>.<span class="w"> </span><span class="nv">This</span><span class="w"> </span><span class="nv">will</span><span class="w"> </span><span class="nv">link</span>
<span class="w"> </span><span class="nv">these</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">files</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="o">/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">null</span>,<span class="w"> </span><span class="nv">making</span><span class="w"> </span><span class="nv">it</span><span class="w"> </span><span class="nv">impossible</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">start</span><span class="w"> </span><span class="nv">them</span>.<span class="w"> </span><span class="nv">This</span>
<span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">stronger</span><span class="w"> </span><span class="nv">version</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">disable</span>,<span class="w"> </span><span class="nv">since</span><span class="w"> </span><span class="nv">it</span><span class="w"> </span><span class="nv">prohibits</span><span class="w"> </span><span class="nv">all</span><span class="w"> </span><span class="nv">kinds</span><span class="w"> </span><span class="nv">of</span>
<span class="w"> </span><span class="nv">activation</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">unit</span>,<span class="w"> </span><span class="nv">including</span><span class="w"> </span><span class="nv">enablement</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="nv">manual</span><span class="w"> </span><span class="nv">activation</span>.<span class="w"> </span><span class="nv">Use</span>
<span class="w"> </span><span class="nv">this</span><span class="w"> </span><span class="nv">option</span><span class="w"> </span><span class="nv">with</span><span class="w"> </span><span class="nv">care</span>.<span class="w"> </span><span class="nv">This</span><span class="w"> </span><span class="nv">honors</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="o">--</span><span class="nv">runtime</span><span class="w"> </span><span class="nv">option</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">only</span><span class="w"> </span><span class="nv">mask</span>
<span class="w"> </span><span class="nv">temporarily</span><span class="w"> </span><span class="k">until</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="k">next</span><span class="w"> </span><span class="nv">reboot</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">system</span>.<span class="w"> </span><span class="nv">The</span><span class="w"> </span><span class="o">--</span><span class="nv">now</span><span class="w"> </span><span class="nv">option</span><span class="w"> </span><span class="nv">may</span><span class="w"> </span><span class="nv">be</span>
<span class="w"> </span><span class="nv">used</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">ensure</span><span class="w"> </span><span class="nv">that</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">units</span><span class="w"> </span><span class="nv">are</span><span class="w"> </span><span class="nv">also</span><span class="w"> </span><span class="nv">stopped</span>.<span class="w"> </span><span class="nv">This</span><span class="w"> </span><span class="nv">command</span><span class="w"> </span><span class="nv">expects</span>
<span class="w"> </span><span class="nv">valid</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">names</span><span class="w"> </span><span class="nv">only</span>,<span class="w"> </span><span class="nv">it</span><span class="w"> </span><span class="nv">does</span><span class="w"> </span><span class="nv">not</span><span class="w"> </span><span class="nv">accept</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">file</span><span class="w"> </span><span class="nv">paths</span>.
<span class="w"> </span><span class="nv">unmask</span><span class="w"> </span><span class="nv">NAME</span>...
<span class="w"> </span><span class="nv">Unmask</span><span class="w"> </span><span class="nv">one</span><span class="w"> </span><span class="nv">or</span><span class="w"> </span><span class="nv">more</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">files</span>,<span class="w"> </span><span class="nv">as</span><span class="w"> </span><span class="nv">specified</span><span class="w"> </span><span class="nv">on</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">command</span><span class="w"> </span><span class="nv">line</span>.<span class="w"> </span><span class="nv">This</span>
<span class="w"> </span><span class="nv">will</span><span class="w"> </span><span class="nv">undo</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">effect</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">mask</span>.<span class="w"> </span><span class="nv">This</span><span class="w"> </span><span class="nv">command</span><span class="w"> </span><span class="nv">expects</span><span class="w"> </span><span class="nv">valid</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">names</span>
<span class="w"> </span><span class="nv">only</span>,<span class="w"> </span><span class="nv">it</span><span class="w"> </span><span class="nv">does</span><span class="w"> </span><span class="nv">not</span><span class="w"> </span><span class="nv">accept</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">file</span><span class="w"> </span><span class="nv">paths</span>.
</pre></div>
<p>So basically we may tell systemd to make sure that even if service is enabled
or started manually it will refuse to do so.</p>
<p>But how that solves problem of auto - starting upon installation? Before we
<strong>apt-get/dnf/yum install</strong> we may mask service we want to install by creating
symlink:</p>
<div class="highlight"><pre><span></span><span class="w"> </span>ln<span class="w"> </span>-s<span class="w"> </span>/dev/null<span class="w"> </span>/etc/systemd/system/apache2.service
</pre></div>
<p>This way Apache will not be started after installation. </p>
<h2>Vendor policy and default presets</h2>
<p>systemd also provides packages vendors with possibility to define if
application should or should not be started upon installation. It's a very
good win - win solution, as now maintainer may actually decide.</p>
<p>If you want to see how service behaves after installation simply:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@fedex<span class="w"> </span>system<span class="o">]</span><span class="c1"># systemctl status docker</span>
●<span class="w"> </span>docker.service<span class="w"> </span>-<span class="w"> </span>Docker<span class="w"> </span>Application<span class="w"> </span>Container<span class="w"> </span>Engine
<span class="w"> </span>Loaded:<span class="w"> </span>loaded<span class="w"> </span><span class="o">(</span>/usr/lib/systemd/system/docker.service<span class="p">;</span><span class="w"> </span>enabled<span class="p">;</span><span class="w"> </span>vendor<span class="w"> </span>preset:<span class="w"> </span>disabled<span class="o">)</span>
</pre></div>
<p>Take not of <strong>vendor preset: disabled</strong> above. It tells us that auto - starting
upon installation is disabled for this package.</p>
<p>systemd also provides a functionality to manage presets with <strong>systemctl</strong>:</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="nv">preset</span><span class="w"> </span><span class="nv">NAME</span>...
<span class="w"> </span><span class="nv">Reset</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">enable</span><span class="o">/</span><span class="nv">disable</span><span class="w"> </span><span class="nv">status</span><span class="w"> </span><span class="nv">one</span><span class="w"> </span><span class="nv">or</span><span class="w"> </span><span class="nv">more</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">files</span>,<span class="w"> </span><span class="nv">as</span><span class="w"> </span><span class="nv">specified</span><span class="w"> </span><span class="nv">on</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">command</span><span class="w"> </span><span class="nv">line</span>,<span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">defaults</span><span class="w"> </span><span class="nv">configured</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">preset</span><span class="w"> </span><span class="nv">policy</span><span class="w"> </span><span class="nv">files</span>.<span class="w"> </span><span class="nv">This</span><span class="w"> </span><span class="nv">has</span><span class="w"> </span><span class="nv">the</span>
<span class="w"> </span><span class="nv">same</span><span class="w"> </span><span class="nv">effect</span><span class="w"> </span><span class="nv">as</span><span class="w"> </span><span class="nv">disable</span><span class="w"> </span><span class="nv">or</span><span class="w"> </span><span class="nv">enable</span>,<span class="w"> </span><span class="nv">depending</span><span class="w"> </span><span class="nv">how</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">listed</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">preset</span><span class="w"> </span><span class="nv">files</span>.
<span class="w"> </span><span class="nv">Use</span><span class="w"> </span><span class="o">--</span><span class="nv">preset</span><span class="o">-</span><span class="nv">mode</span><span class="o">=</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">control</span><span class="w"> </span><span class="nv">whether</span><span class="w"> </span><span class="nv">units</span><span class="w"> </span><span class="nv">shall</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">enabled</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="nv">disabled</span>,<span class="w"> </span><span class="nv">or</span><span class="w"> </span><span class="nv">only</span><span class="w"> </span><span class="nv">enabled</span>,<span class="w"> </span><span class="nv">or</span><span class="w"> </span><span class="nv">only</span><span class="w"> </span><span class="nv">disabled</span>.
<span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">carries</span><span class="w"> </span><span class="nv">no</span><span class="w"> </span><span class="nv">install</span><span class="w"> </span><span class="nv">information</span>,<span class="w"> </span><span class="nv">it</span><span class="w"> </span><span class="nv">will</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">silently</span><span class="w"> </span><span class="nv">ignored</span><span class="w"> </span><span class="nv">by</span><span class="w"> </span><span class="nv">this</span><span class="w"> </span><span class="nv">command</span>.<span class="w"> </span><span class="nv">NAME</span><span class="w"> </span><span class="nv">must</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">real</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">name</span>,<span class="w"> </span><span class="nv">any</span><span class="w"> </span><span class="nv">alias</span><span class="w"> </span><span class="nv">names</span><span class="w"> </span><span class="nv">are</span><span class="w"> </span><span class="nv">ignored</span>
<span class="w"> </span><span class="nv">silently</span>.
<span class="w"> </span><span class="k">For</span><span class="w"> </span><span class="nv">more</span><span class="w"> </span><span class="nv">information</span><span class="w"> </span><span class="nv">on</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">preset</span><span class="w"> </span><span class="nv">policy</span><span class="w"> </span><span class="nv">format</span>,<span class="w"> </span><span class="nv">see</span><span class="w"> </span><span class="nv">systemd</span>.<span class="nv">preset</span><span class="ss">(</span><span class="mi">5</span><span class="ss">)</span>.<span class="w"> </span><span class="k">For</span><span class="w"> </span><span class="nv">more</span><span class="w"> </span><span class="nv">information</span><span class="w"> </span><span class="nv">on</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">concept</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">presets</span>,<span class="w"> </span><span class="nv">please</span><span class="w"> </span><span class="nv">consult</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">Preset</span>[<span class="mi">1</span>]
<span class="w"> </span><span class="nv">document</span>.
<span class="w"> </span><span class="nv">preset</span><span class="o">-</span><span class="nv">all</span>
<span class="w"> </span><span class="nv">Resets</span><span class="w"> </span><span class="nv">all</span><span class="w"> </span><span class="nv">installed</span><span class="w"> </span><span class="nv">unit</span><span class="w"> </span><span class="nv">files</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">defaults</span><span class="w"> </span><span class="nv">configured</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">preset</span><span class="w"> </span><span class="nv">policy</span><span class="w"> </span><span class="nv">file</span><span class="w"> </span><span class="ss">(</span><span class="nv">see</span><span class="w"> </span><span class="nv">above</span><span class="ss">)</span>.
<span class="w"> </span><span class="nv">Use</span><span class="w"> </span><span class="o">--</span><span class="nv">preset</span><span class="o">-</span><span class="nv">mode</span><span class="o">=</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">control</span><span class="w"> </span><span class="nv">whether</span><span class="w"> </span><span class="nv">units</span><span class="w"> </span><span class="nv">shall</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">enabled</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="nv">disabled</span>,<span class="w"> </span><span class="nv">or</span><span class="w"> </span><span class="nv">only</span><span class="w"> </span><span class="nv">enabled</span>,<span class="w"> </span><span class="nv">or</span><span class="w"> </span><span class="nv">only</span><span class="w"> </span><span class="nv">disabled</span>
</pre></div>
<h2>Anything more? #learningsystemd!</h2>
<p>If you wanna learn more about <strong>systemd</strong> simply follow
<a href="/feeds/tag/learning-systemd.rss.xml">learning-systemd RSS</a>
or
<a href="/feeds/tag/learning-systemd.atom.xml">learning-systemd ATOM</a>
tag in this very blog (or on Twitter:
<a href="https://twitter.com/search?f=tweets&q=%23learningsystemd&src=typd">#learningsystemd</a>)</p>systemd workshops recap2016-11-27T00:00:00+01:002016-11-27T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-11-27:2016/Nov/27/systemd-workshops-recap/<p><center><img alt="PID1 for the win!" src="https://maciej.lasyk.info/images/pid1.png" /></center></p>
<h2>Workshops</h2>
<p>So as <a href="https://maciej.lasyk.info/2016/Nov/15/systemd-workshops/">I wrote before</a>
together with <a href="https://twitter.com/marcinskarbek">Marcin Skarbek</a> we
ran <strong>systemd</strong> workshops last Saturday. Venue was full, guys from
<a href="http://www.megiteam.pl/">MegiTeam</a> made sure everything was organized well -
it was just like it should be - thank you all :)</p>
<p><center><img alt="systemd workshops" src="https://maciej.lasyk.info/images/systemd-workshops-krk.jpg" /></center></p>
<h2>Can I see what you presented?</h2>
<p>Sure, everything is explained in <a href="https://github.com/docent-net/systemd-workshop">github repo</a>.</p>
<h2>Will you do it again?</h2>
<p>There've been questions if we could run those workshops one more time. So
actually yes. We're thinking about 2nd round before end of this year. Simply
follow Twitter tag <a href="https://twitter.com/search?q=%23systemdkrkworkshops">#systemdkrkworkshops</a>
(or this blog's RSSes).</p>
<h2>Anything more? #learningsystemd!</h2>
<p>Yes. Considering that systemd is huge and learning it is a long journey I
decided to create <strong>learning-systemd</strong> tag in this very blog (and Twitter tag
<a href="https://twitter.com/search?f=tweets&q=%23learningsystemd&src=typd">#learningsystemd</a>)
I'll use it for blog post I'll create on this very topic. And I plan to post
often :)</p>Devops meetup @Wrocław2016-11-21T00:00:00+01:002016-11-21T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-11-21:2016/Nov/21/devops-meetup-wroclaw/<p><center><img alt="PID1 for the win!" src="https://maciej.lasyk.info/images/pid1.png" /></center></p>
<h2>Meetup</h2>
<p>I was asked to give a 90min presentation which be a very short version of
<a href="https://github.com/docent-net/systemd-workshop">systemd workshops I'm running</a>
here in Kraków.</p>
<p>If you're interested in this very subject -
<a href="https://www.meetup.com/Wroclaw-DevOps-Meetup/events/235730817/?fromEmail=235730817&rv=ea1">register now on the meetup page</a></p>
<p>See you there guys!</p>Devopsdays Warsaw2016-11-19T00:00:00+01:002016-11-19T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-11-19:2016/Nov/19/devopsdays-warsaw/<p><center><img alt="Devopsdays Warsaw" src="https://maciej.lasyk.info/images/devopsdays-logo.png" /></center></p>
<h2>Get yourself there!</h2>
<p>Once again we're hosting <a href="https://www.devopsdays.org/">devopsdays event</a> in
Warsaw, Poland, thanks to <a href="http://proidea.pl/">Proidea</a> that takes care of
organization.</p>
<p>I'm member of the programme committee - that means, that I had chance to work
with great guys on reading, accepting and filtering talk proposals as well
as setting up the final schedule.</p>
<h4>Schedule</h4>
<p>So what'll be there? Actually <a href="http://2016.devopsdays.pl/program/schedule/">you should see yourself</a>
- the schedule is full of various talks. Thanks to that attendees will be able
to expand their knowledge - following the true devops learning path.</p>
<h4>Ignite talks</h4>
<p>If you feel like adding something, you wanna to show something - you may submit
<a href="http://2016.devopsdays.pl/cfp/ignites/form/">an ignite talk</a> so you'll have
your 5 minutes on the stage. Great opportunity for all of you that something
to say.</p>
<h4>Open spaces</h4>
<p>We'll organize also <a href="http://2016.devopsdays.pl/open-spaces/">open - spaces discussions</a>.
Idea is that you write down on a white - board subject / problem that you'd
like to discuss with others, then those proposals are voted and those with
most number of votes are chosen. Later we gather in small groups discussing
upvoted topics. </p>
<h4>Registration</h4>
<p><a href="http://2016.devopsdays.pl/faq/conference/">Registration is still possible.</a></p>
<h4>FAQ</h4>
<p>There's also <a href="http://2016.devopsdays.pl/faq/conference/">FAQ section</a> where
you may find answers for not that obvious questions.</p>
<h4>Parties!</h4>
<p>Last but not least - we'll <a href="http://2016.devopsdays.pl/news/15/">have a before and after - parties</a>.
Actually those events are best place for meeting and talking to speakers about
anything. Speakers usually open themselves during those kind of parties, so
don't loose your chance! :)</p>
<p>See you there guys!</p>systemd workshops2016-11-15T00:00:00+01:002016-11-15T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-11-15:2016/Nov/15/systemd-workshops/<p><center><img alt="PID1 for the win!" src="https://maciej.lasyk.info/images/pid1.png" /></center></p>
<h2>The workshops</h2>
<p>Since my last attendance to <a href="https://conf.systemd.io/">systemd conference</a>
together with <a href="https://twitter.com/marcinskarbek">Marcin Skarbek</a> we've been
thinking about running a systemd workshops in Kraków.</p>
<p>So actually during last <a href="https://www.meetup.com/AnonimowiAdmini/">AA meetup</a>
we finally decided (Thanks Zuza!) to give it a go.</p>
<p>So it is! Probably on 26th November we'll host all - day hands - on workshops
in Kraków.</p>
<p>For more details see <a href="https://github.com/docent-net/systemd-workshop">workshop Github repository</a>
or follow Twitter <a href="https://twitter.com/search?q=%23systemdkrkworkshops">#systemdkrkworkshops</a>
tag.</p>
<p>See you there guys!</p>Working on blog2016-11-14T00:00:00+01:002016-11-14T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-11-14:2016/Nov/14/working-on-blog/<p><center><img alt="boring" src="https://maciej.lasyk.info/images/boring.jpg" /></center></p>
<h2>Changes (again?...)</h2>
<p>Yes, number of "todos" in my Workflowy list according to this blog reached a
level, when I just have to start working on it. There're many things:</p>
<ol>
<li>Fix layout:
1, It has to work fine on mobile as well as on 27"</li>
<li>It has to be nice, readable instead of contrasting and heavy</li>
<li>It has to load very fast (under 100ms)</li>
<li>No cumbersome clicking</li>
<li>Finally host with SSL</li>
<li>Add URLs for Github, Twitter, StackOverflow etc (contact page)</li>
<li>Create 3 main categories (tech, sport, random) with RSS per each (and one
global)</li>
<li>Add tag + section of my books library (all categories)</li>
<li>Import posts from http://maciek.lasyk.info, close old page and 301 it!</li>
</ol>
<p>What I actually done already?</p>
<ol>
<li>Fix fonts</li>
<li>Fix not showing tags after Pelican upgrade</li>
<li>Fix deployment model (own bare repo w/post-receive hook deploying to prod +
repo mirror on github)</li>
</ol>
<p>So now actually you may find this blog's whole source <a href="https://github.com/docent-net/maciej.lasyk.info">on github</a>!
Not a big deal, but maybe for some of you it's gonna be interesting.</p>
<p>Also I started working on importer script that will import old Wordpress
entries to this blog (see on <a href="maciej.lasyk.info/wordpress_export/import.py">Github</a></p>systemdconf(2): 1st day / workshop summary2016-09-29T00:00:00+02:002016-09-29T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2016-09-29:2016/Sep/29/systemdconf2-1st-day-workshop-summary/<p><center><img alt="PID1" src="https://maciej.lasyk.info/images/pid1.png" /></center></p>
<h2>systemdconf(2)</h2>
<p>So it's the second time <strong>systemdconf</strong> takes place. Again - in Berlin, which I find even more pretty in early Autumn
than year before (which was in November) - sunny and warm.</p>
<p>Full schedule of this year's edition is <a href="https://cfp.systemd.io/en/systemdconf_2016/public/schedule/0">here</a>. Last year
there was no workshops, but there was a hackfest on the last day. This time we have a workshop on the 0 - day (I call
it "0 - day" as this is an optional and needs an additional ticket).</p>
<p>So first 2 sessions were common for all attendees:</p>
<ul>
<li><strong>Demystifying systemd - Hands On</strong> by <strong>Ben Breard</strong></li>
<li><strong>Using the Journal Efficiently</strong> by <strong>Lennart Poettering</strong></li>
</ul>
<p>Afterwards I found it very hard to choose between next sessions that were taking place in same time. I really wanted
to take part in <strong>Lukáš Nykrýn's</strong> workshop about <strong>systemd-nspawn 101</strong> and learn about <strong>mgmt</strong> from <strong>James Shubin</strong>.
But I finally chose almost 5 - hour session about <strong>Writing services the systemd way</strong> lead by <strong>David Strauss</strong>.</p>
<h2>Session 1: Demystifying systemd - Hands on by Ben Breard</h2>
<p>I was late almost 90 minutes as my coach bus was late almost 2 - hours (thanks to traffic jams on south entrance to
Berlin). But I quickly synced to the rest of people thanks to <a href="http://people.redhat.com/bbreard/systemd/DemystifyingsystemdHandsOn.html">wonderful materials prepared by
Ben</a>.</p>
<p>I decided to go through this workshop without using VMs but systemd-nspawn container instead. Thanks to this approach
I was able to create one container per workshop task in a quick and elegant way without wasting resources on KVM
VMs or some other Vagrant abomination ;)</p>
<p>I found a little error in the materials for systemd-nspawn preparation. It should be:</p>
<div class="highlight"><pre><span></span><span class="n">dnf</span><span class="w"> </span><span class="o">--</span><span class="n">releasever</span><span class="o">=</span><span class="mi">24</span><span class="w"> </span><span class="o">--</span><span class="n">installroot</span><span class="o">=/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">machine</span><span class="o">/</span><span class="n">f24</span><span class="w"> </span>\
<span class="n">install</span><span class="w"> </span><span class="n">systemd</span><span class="w"> </span><span class="n">passwd</span><span class="w"> </span><span class="n">dnf</span><span class="w"> </span><span class="n">fedora</span><span class="o">-</span><span class="n">release</span>
</pre></div>
<p>Also I suggest creating this container under <strong>/var/lib/machines</strong> as this directory already has proper selinux type
(and we don't want to run with selinux disabled, yeah?)</p>
<p>Also I'd suggest installing following packages:</p>
<div class="highlight"><pre><span></span>dnf install systemd-devel telnet telnet-server procps \
pkg-config make automake gcc git vim-enhanced iproute
</pre></div>
<p>And in the end - use this container as template for any following task from all the workshop sessions:</p>
<div class="highlight"><pre><span></span><span class="n">cp</span><span class="w"> </span><span class="o">-</span><span class="n">R</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">machines</span><span class="o">/</span><span class="n">f24</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">machines</span><span class="o">/</span><span class="n">f24</span><span class="o">-</span><span class="n">workshop1</span>
</pre></div>
<p>I've published notes from this session on <a href="https://github.com/docent-net/systemd-conference-2016">my github</a></p>
<p>Generally speaking I'm very glad to attend this session - it was a wholesome recap of systemd basics!</p>
<h2>Session 2: Using the Journal Efficiently by Lennart Poettering</h2>
<p>This session was run by Lennart. If I were asked <strong>how to describe it in one sentence</strong> it would be: <strong>man systemctl
with Lennart's incredible comment and things that can't be find in man</strong>. I really loved it. And as usual - Lennart
explained the matter of logging w/journald so extensively that actually it was hard to even ask about anything.</p>
<p>Beside <strong>man systemctl</strong> things Lennart talked in details about unit's metadata (see <strong>-o verbose</strong> and all those
uppercase params), cursors and gatewayd.</p>
<p>I've published notes from this session on <a href="https://github.com/docent-net/systemd-conference-2016">my github</a></p>
<p>Again - outstanding session.</p>
<h2>Session 3: Writing services the systemd way by David Strauss</h2>
<p>During this session we created a simple C application which later we converted into a systemd service with some
basic systemd functionalities:</p>
<ul>
<li>journald integration via sd_journal</li>
<li>monitoring / watchdog via sd_notify</li>
<li>socket activation </li>
</ul>
<p>Whole session was packed with coding, compiling, re - compiling and trying. It was very intensive, fast - paced and...
great! Afterwards we also saw corresponding <a href="https://github.com/systemd/python-systemd">python-systemd</a> wrappers.
Actually one of my ideas for Saturday hackfest is to rewrite this session's subject in Python in order to learn
how Python integration works.</p>
<p>Again - totally awesome session!</p>
<p>Materials for this session are hosted on <a href="https://github.com/systemd/tutorials/tree/berlin">Github</a>. Simply start
with <strong>starter</strong> directory and apply logdaemon, watchdogdaemon and sadaemon on that starter content.</p>
<p>I didn't make notes during this session - there was no time, only coding and compiling.</p>
<h2>systemd-nspawn 101 by Lukáš Nykrýn</h2>
<p>Unfortunately I didn't attend this workshops. However I asked Lukáš about materials -
<a href="http://redhat.slides.com/lnykryn/systemd-nspawn-101#">he already published his slides</a></p>
<h2>Next Generation Config Mgmt: Workshop by James (purpleidea) Shubin</h2>
<p>And now for something completely different ;) Also - I couldn't attend this one, but I caught up day after with James
and asked about materials. However there're no dedicated materials from that workshops, you can basically hack around
mgmt basing on <a href="https://github.com/purpleidea/mgmt/">github project page</a>. This is very interesting project which needs
contributions (GoLang here) so if you're interested - go for it!</p>
<h2>So that's it</h2>
<p>Basically that's it. I have to say that I really like the idea of workshops and this day was 100% worth its price. I
was able to recap my systemd knowledge, update it, ask questions to people who actually develop systemd and understand
concepts which were not tha easy for me. Also I already have some implementations ideas that were born during those
workshops so - thank you guys!</p>Handling timeouts in Ansible AWS modules2016-06-28T00:00:00+02:002016-06-28T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2016-06-28:2016/Jun/28/handling-timeouts-in-ansible-aws-modules/<p><center><img alt="AWS" src="https://maciej.lasyk.info/images/aws_logo.png" /></center></p>
<h2>Timeouts in AWS?</h2>
<p>Yup, it's completely normal, that sometimes between your laptop and AWS API something
doesn't work as stable as you'd like it to. We hit random timeouts and we have to
re - run our playbooks.</p>
<h2>Timeouts in Ansible?</h2>
<p>Looking for any "timeout" configuration directive in ansible.cfg <a href="http://docs.ansible.com/ansible/intro_configuration.html#timeout">I found this</a> -
it actually doesn't help. We might just set a longer timeout for SSH connections. But
AWS modules connects to API using boto library.</p>
<h2>So what's the problem once again?</h2>
<p>Let's say that we have the following playbook:</p>
<div class="highlight"><pre><span></span><span class="x">- hosts: localhost</span>
<span class="x"> connection: local</span>
<span class="x"> tasks:</span>
<span class="x"> - name: Create security group</span>
<span class="x"> ec2_group:</span>
<span class="x"> name: "test_sg"</span>
<span class="x"> state: "present"</span>
<span class="x"> vpc_id: "some_vpc"</span>
<span class="x"> purge_rules: False</span>
<span class="x"> purge_rules_egress: False</span>
<span class="x"> register: ec2_group</span>
<span class="x"> - name: Tag security group</span>
<span class="x"> ec2_tag:</span>
<span class="x"> resource: "</span><span class="cp">{{</span> <span class="nv">ec2_group.group_id</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> state: "present"</span>
<span class="x"> tags: "</span><span class="cp">{{</span> <span class="nv">some_defined_tags</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> register: ec2_group_tags</span>
<span class="x"> - name: Apply permissions on security group</span>
<span class="x"> ec2_group:</span>
<span class="x"> name: "test_sg"</span>
<span class="x"> state: "present"</span>
<span class="x"> vpc_id: "some_vpc"</span>
<span class="x"> rules: "</span><span class="cp">{{</span> <span class="nv">some</span><span class="o">)</span><span class="nv">defined_rules</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> - name: Provision a set of instances</span>
<span class="x"> ec2:</span>
<span class="x"> group_id: "</span><span class="cp">{{</span> <span class="nv">ec2_group.group_id</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> instance_type: "t2.large"</span>
<span class="x"> image: "some_ami"</span>
<span class="x"> vpc_subnet_id: "some_vpc"</span>
<span class="x"> count_tag: "some_tag"</span>
<span class="x"> exact_count: "3"</span>
<span class="x"> wait: true</span>
<span class="x"> instance_tags: "</span><span class="cp">{{</span> <span class="nv">some_defined_tags</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> zone: "some_zone"</span>
<span class="x"> register: ec2</span>
</pre></div>
<p>And now let's say that it fails randomly because of timeouts that happens over time -
and those timeouts hit random tasks.</p>
<p>We could simply re - run the job and thanks to idempotency it would just make sure
that it was all finished up.</p>
<p>Also we could simply retry starting from failing step:</p>
<div class="highlight"><pre><span></span>PLAY RECAP ********************************************************************
to retry, use: --limit @/home/user/playbook.retry
</pre></div>
<h2>Anything more civilized?</h2>
<p>Actually there's another method. Not sure if it is more civilized, but it works
and makes the Ansible playbook finish successfully more frequently.</p>
<p>It's a simple <strong>do - until</strong> loop <a href="http://docs.ansible.com/ansible/playbooks_loops.html#do-until-loops">documented here</a></p>
<p>Using this approach we might add error - handling and retry policy to above playbook
so we could get something like the following one;</p>
<div class="highlight"><pre><span></span><span class="x">- hosts: localhost</span>
<span class="x"> connection: local</span>
<span class="x"> tasks:</span>
<span class="x"> - name: Create security group</span>
<span class="x"> ec2_group:</span>
<span class="x"> name: "test_sg"</span>
<span class="x"> state: "present"</span>
<span class="x"> vpc_id: "some_vpc"</span>
<span class="x"> purge_rules: False</span>
<span class="x"> purge_rules_egress: False</span>
<span class="x"> register: ec2_group</span>
<span class="x"> until: ec2_group.failed is not defined or ec2_group.failed == false</span>
<span class="x"> retries: "3"</span>
<span class="x"> delay: "30"</span>
<span class="x"> - name: Tag security group</span>
<span class="x"> ec2_tag:</span>
<span class="x"> resource: "</span><span class="cp">{{</span> <span class="nv">ec2_group.group_id</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> state: "present"</span>
<span class="x"> tags: "</span><span class="cp">{{</span> <span class="nv">some_defined_tags</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> register: ec2_group_tags</span>
<span class="x"> until: ec2_group_tags.failed is not defined or ec2_group_tags.failed == false</span>
<span class="x"> retries: "3"</span>
<span class="x"> delay: "30"</span>
<span class="x"> - name: Apply permissions on security group</span>
<span class="x"> ec2_group:</span>
<span class="x"> name: "test_sg"</span>
<span class="x"> state: "present"</span>
<span class="x"> vpc_id: "some_vpc"</span>
<span class="x"> rules: "</span><span class="cp">{{</span> <span class="nv">some</span><span class="o">)</span><span class="nv">defined_rules</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> register: ec2_group_perms</span>
<span class="x"> until: ec2_group_perms.failed is not defined or ec2_group_perms.failed == false</span>
<span class="x"> retries: "3"</span>
<span class="x"> delay: "30"</span>
<span class="x"> - name: Provision a set of instances</span>
<span class="x"> ec2:</span>
<span class="x"> group_id: "</span><span class="cp">{{</span> <span class="nv">ec2_group.group_id</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> instance_type: "t2.large"</span>
<span class="x"> image: "some_ami"</span>
<span class="x"> vpc_subnet_id: "some_vpc"</span>
<span class="x"> count_tag: "some_tag"</span>
<span class="x"> exact_count: "3"</span>
<span class="x"> wait: true</span>
<span class="x"> instance_tags: "</span><span class="cp">{{</span> <span class="nv">some_defined_tags</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> zone: "some_zone"</span>
<span class="x"> register: ec2</span>
<span class="x"> until: ec2.failed is not defined or ec2.failed == false</span>
<span class="x"> retries: "3"</span>
<span class="x"> delay: "30"</span>
</pre></div>
<p>It's very simple and works just fine. Of course retry and delay values are good subject
to be parametrized and put into variable.</p>Working with virtualenv and Ansible2016-06-27T00:00:00+02:002016-06-27T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2016-06-27:2016/Jun/27/working-with-virtualenv-and-ansible/<p><center><img alt="Evernote" src="https://maciej.lasyk.info/images/Ansible_Logo.png" /></center></p>
<h2>What this is about?</h2>
<p>So after upgrading to <a href="https://getfedora.org/?wkfpF24">Fedora 24</a> I wanted to have
a clean installation and use only system libraries (no more <strong>sudo pip install..</strong>).
I already achieved that by using
<a href="https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html">systemd-nspawn containers</a>
for application isolation (instead of using virtual machines) and <a href="http://docs.python-guide.org/en/latest/dev/virtualenvs/">virtualenv</a>
for Python libraries during pydevelopment.</p>
<p>But.. while working with Ansible there's still a problem. E.g. when working with cloud
providers (like AWS, Google, Rackspace, Digital Ocean - you name it) you usually use
local connections for sending API requests, e.g.:</p>
<div class="highlight"><pre><span></span>- hosts: localhost
connection: local
tasks:
- name: Provision a set of instances
ec2:
...
</pre></div>
<p>In order to make above working you need to have <strong>boto</strong> and <strong>Ansible</strong> installed on
your box. That's a no - brainer.</p>
<p>But what if every ansible repository contains requirements.txt which declares specific
Ansible and boto versions? E.g.:</p>
<div class="highlight"><pre><span></span>ansible==2.0.1.0
boto==2.40.0
</pre></div>
<p>Yes, you could simply:</p>
<div class="highlight"><pre><span></span>sudo pip install ansible==2.0.1.0
sudo pip install boto==2.40.0
</pre></div>
<p>But this would actually break your system - wide libraries. If another project needs
<strong>ansible==2.1.0.0</strong> then you're in trouble.</p>
<h2>Ansible and virtualenv?</h2>
<p>So - the solution is to use <a href="http://docs.python-guide.org/en/latest/dev/virtualenvs/">Python virtualenv</a>.
It might be obvious, but it's not that easy with Ansible. When working from defined virtual environment
and using <strong>local</strong> connection Ansible actually creates SSH connection via loopback device and enters login shell
using configured account (e.g. <strong>remote_user</strong> set it ansible.cfg; more on this topic <a href="http://docs.ansible.com/ansible/intro_configuration.html#remote-user">here</a>).
After logging back into your account we're using default Python interpreter (global one, probably <strong>/usr/bin/python</strong>) -
not the one we want to (from our virtualenv). Kaboom!</p>
<p>Ok, so there's a solution. It's called <strong>ansible_python_interpreter</strong> and it's mentioned
in <a href="http://docs.ansible.com/faq.html">Ansible FAQ</a> (but regarding other case). Basically
if we tell Ansible to use Python interpreter from our virtualenv it will work:</p>
<div class="highlight"><pre><span></span>ansible-playbook -i inventory/ec2.py magic_cloud.yml \
-e "ansible_python_interpreter=/home/ukleftue/.virtualenvs/ilovebaleares/bin/python"
</pre></div>
<p>In above example virtualenv we use is called <strong>ilovebaleares</strong>.</p>
<p>So using this hack'ish way we can marry Ansible and virtualenvs together.</p>
<h2>There's more to it!</h2>
<p>Remember that you can have couple of plays in playbook? Imagine, that first play
actually spawns instances (using API calls invoked from local tasks) and second one
applies roles using direct SSH connections to remote (freshly spawned) instances:</p>
<div class="highlight"><pre><span></span><span class="x">- hosts: localhost</span>
<span class="x"> connection: local</span>
<span class="x"> tasks:</span>
<span class="x"> - name: Provision a set of instances</span>
<span class="x"> ec2:</span>
<span class="x"> ...</span>
<span class="x"> register: ec2</span>
<span class="x"> - name: Add host to temporary group</span>
<span class="x"> add_host: ></span>
<span class="x"> hostname="</span><span class="cp">{{</span> <span class="nv">item.private_ip</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> groupname="tag_awesomehosts"</span>
<span class="x"> with_items: "</span><span class="cp">{{</span> <span class="nv">ec2.tagged_instances</span> <span class="cp">}}</span><span class="x">"</span>
<span class="x"> changed_when: false</span>
<span class="x">- name: Configure instances</span>
<span class="x"> hosts:</span>
<span class="x"> - "tag_awesomehosts"</span>
<span class="x"> roles:</span>
<span class="x"> - do_something</span>
</pre></div>
<p>So in above example using <strong>ansible_python_interpreter</strong> will work for first play,
but for second it will cause:</p>
<p><strong>fatal: [some_ip_addr]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", "unreachable": true}</strong></p>
<p>Why is that? If we run ansible-playbook in verbose mode (add -vvvvv) we'll get:</p>
<div class="highlight"><pre><span></span><span class="o"><</span><span class="nt">some_ip_addr</span><span class="o">></span><span class="w"> </span><span class="nt">SSH</span><span class="o">:</span><span class="w"> </span><span class="nt">EXEC</span><span class="w"> </span><span class="nt">ssh</span><span class="w"> </span><span class="nt">-C</span><span class="w"> </span><span class="nt">-vvv</span><span class="w"> </span><span class="nt">-o</span><span class="w"> </span><span class="nt">ControlMaster</span><span class="o">=</span><span class="nt">auto</span>
<span class="nt">-o</span><span class="w"> </span><span class="nt">ControlPersist</span><span class="o">=</span><span class="nt">60s</span><span class="w"> </span><span class="nt">-o</span><span class="w"> </span><span class="nt">StrictHostKeyChecking</span><span class="o">=</span><span class="nt">no</span><span class="w"> </span><span class="nt">-o</span>
<span class="nt">KbdInteractiveAuthentication</span><span class="o">=</span><span class="nt">no</span><span class="w"> </span><span class="nt">-o</span>
<span class="nt">PreferredAuthentications</span><span class="o">=</span><span class="nt">gssapi-with-mic</span><span class="o">,</span><span class="nt">gssapi-keyex</span><span class="o">,</span><span class="nt">hostbased</span><span class="o">,</span>
<span class="nt">publickey</span><span class="w"> </span><span class="nt">-o</span><span class="w"> </span><span class="nt">PasswordAuthentication</span><span class="o">=</span><span class="nt">no</span><span class="w"> </span><span class="nt">-o</span><span class="w"> </span><span class="nt">User</span><span class="o">=</span><span class="nt">ec2-user</span>
<span class="nt">-o</span><span class="w"> </span><span class="nt">ConnectTimeout</span><span class="o">=</span><span class="nt">20</span><span class="w"> </span><span class="nt">-o</span><span class="w"> </span><span class="nt">ControlPath</span><span class="o">=/</span><span class="nt">tmp</span><span class="o">/</span><span class="nt">ansible-ssh-</span><span class="o">%</span><span class="nt">h-</span><span class="o">%</span><span class="nt">p-</span><span class="o">%</span><span class="nt">r</span><span class="w"> </span><span class="nt">some_ip_addr</span>
<span class="s1">'/bin/sh -c '</span><span class="s2">"'"</span><span class="s1">'sudo -H -S -n -u root /bin/sh -c '</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'echo</span>
<span class="s1">BECOME-SUCCESS; /bin/sh -c '</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'</span>
<span class="s1">LANG=C LC_ALL=C LC_MESSAGES=C</span>
<span class="s1">/home/ukleftue//.virtualenvs/ilovebaleares/bin/python'</span>
<span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">''</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">'"'</span><span class="s2">"'"</span><span class="s1">''</span><span class="s2">"'"</span><span class="s1">''</span>
</pre></div>
<p>Notice last: <strong>/home/ukleftue//.virtualenvs/ilovebaleares/bin/python</strong> - see? This is
our problem - setting <strong>ansible_python_interpreter</strong> works globally for all plays
within playbook. In the second playbook Ansible connects to freshly spawned EC2 instance
and tries to work under Python interpreter under defined virtualenv - which obviously
doesn't exist on this fresh VM.</p>
<p>So I found solution for this - actually you can set <strong>ansible_python_interpreter</strong> per
host basis in inventory (told you that sometimes when I don't hate Ansible I really
love it?). So just prepare yourself file like <strong>inventory/local</strong> which could look
like this:</p>
<div class="highlight"><pre><span></span><span class="k">[localhost]</span>
<span class="na">localhost ansible_python_interpreter</span><span class="o">=</span><span class="s">/home/ukleftue/.virtualenvs/ilovebaleares/bin/python</span>
</pre></div>
<p>and run above playbook without passing directly <strong>ansible_python_interpreter</strong> on command
line (as it is already set in inventory), but providing additional inventory file:</p>
<div class="highlight"><pre><span></span>ansible-playbook -i inventory/ec2.py -i inventory/local magic_cloud.yml"
</pre></div>
<p>Thanks to that you can define Python interpreter (namely your virtualenv) for local
connections and leave it default for remote connections.</p>
<h2>Whoah, that really works?</h2>
<p>Erm, SOA#1: Works for me ;)</p>
<p>Enjoy!</p>Nixnote (Evernote client) on Fedora2016-03-13T00:00:00+01:002016-03-13T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2016-03-13:2016/Mar/13/nixnote-evernote-client-on-fedora/<p><center><img alt="Evernote" src="https://maciej.lasyk.info/images/evernote_logo.png" /></center></p>
<p>I've been using <a href="http://www.evernote.com">Evernote</a> for years now (Grzegorz, still
remember, thanks!). What I've been missing since 1st day was a decent, Linux native
client. That's why I've been using webapp (which is a really nice piece of work, but
sometimes performs terribly).</p>
<p>There's always been this <a href="http://nevernote.sourceforge.net/">Nevernote</a> app, which was
a desktop open-source client for Evernote, later called <strong>Nixnote</strong> and project webpage
was moved <a href="https://sourceforge.net/projects/nevernote/?source=navbar">here</a>.</p>
<p>Back then version 1.x was developed in Java and it's performance was even more
terrible than native Evernote webapp. But since couple of months we've got a solution
- guys created 2.x version which is based on QT and performs really good. It's still in
beta however but "so far so good" :)</p>
<p>As usual I created ansible-role for this Nixnote - it's very simple - you can check it
in my <a href="https://github.com/docent-net/fedora-desktop-ansible">fedora-desktop-ansible</a> project
on github.</p>Fixing text highlighting in tmux/man/less2015-12-21T00:00:00+01:002015-12-21T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-12-21:2015/Dec/21/fixing-text-highlighting-in-tmuxmanless/<p><center><img alt="tmux" src="https://maciej.lasyk.info/images/tmux.png" /></center></p>
<p>So today I spotted an in issue while using search function when reading man
pages. Basically <strong>I didn't have results highlighted</strong>, but search worked
(position of screen scrolled to somewhere near first spotted result).</p>
<p>Firstly tried to blame <a href="https://en.wikipedia.org/wiki/Terminator_%28terminal_emulator%29">Terminator</a>
but I had same issue in <a href="https://en.wikipedia.org/wiki/Guake">Guake</a> session. Finally I found that
the common denominator here was tmux. Outside of tmux there was no such problems.</p>
<h3>Solution?</h3>
<p>So I dug the internets and <a href="http://tmux.svn.sourceforge.net/viewvc/tmux/trunk/FAQ">found a solution</a>:</p>
<p><em>vim displays reverse video instead of italics, while less displays italics
(or just regular text) instead of reverse. What's wrong?</em></p>
<p><em>Screen's terminfo description lacks italics mode and has standout mode in its
place, but using the same escape sequence that urxvt uses for italics. This
means applications (like vim) looking for italics will not find it and might
turn to reverse in its place, while applications (like less) asking for
standout will end up with italics instead of reverse. To make applications
aware that tmux supports italics and to use a proper escape sequence for
standout, you'll need to create a new terminfo file with modified sgr, smso,
rmso, sitm and ritm entries:</em></p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>mkdir<span class="w"> </span><span class="nv">$HOME</span>/.terminfo/
$<span class="w"> </span><span class="nv">screen_terminfo</span><span class="o">=</span><span class="s2">"screen"</span>
$<span class="w"> </span>infocmp<span class="w"> </span><span class="s2">"</span><span class="nv">$screen_terminfo</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/^screen[^|]*|[^,]*,/screen-it|screen with italics support,/'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/%?%p1%t;3%/%?%p1%t;7%/'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/smso=[^,]*,/smso=\\E[7m,/'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/rmso=[^,]*,/rmso=\\E[27m,/'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'$s/$/ sitm=\\E[3m, ritm=\\E[23m,/'</span><span class="w"> </span>><span class="w"> </span>/tmp/screen.terminfo
$<span class="w"> </span>tic<span class="w"> </span>/tmp/screen.terminfo
</pre></div>
<p><em>And tell tmux to use it in ~/.tmux.conf:</em></p>
<div class="highlight"><pre><span></span>set -g default-terminal "screen-it"
</pre></div>
<p><em>If your terminal supports 256 colors, use:</em></p>
<div class="highlight"><pre><span></span>$<span class="w"> </span><span class="nv">screen_terminfo</span><span class="o">=</span><span class="s2">"screen-256color"</span>
</pre></div>
<p><em>instead of "screen".</em></p>
<h3>Ansible playbook for this</h3>
<p>So I created <a href="https://github.com/docent-net/fedora-desktop-ansible/commit/d314aedd61e6973df3de16bd557de7e3f3aecb3d">Ansible implementation</a>
and put in inside of <a href="https://github.com/docent-net/fedora-desktop-ansible">fedora-desktop-ansible</a>.</p>
<p>You can test it with:</p>
<div class="highlight"><pre><span></span>ansible-playbook<span class="w"> </span>master.yml<span class="w"> </span>--tags<span class="w"> </span>tmux
</pre></div>Containers do not contain2015-11-27T00:00:00+01:002015-11-27T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-11-27:2015/Nov/27/containers-do-not-contain/<p><center><img alt="Devopsdays" src="https://maciej.lasyk.info/images/devopsdays-logo.png" /></center></p>
<p>The title of this presentation is borrowed from <a href="https://www.youtube.com/watch?v=zWGFqMuEHdw">Dan's Walsh</a>
presentations about security of Linux containers.</p>
<p>I worked on this presentation only for about 45 minutes. It was an emergency one - one of the speakers couldn't make it
to the Devopsdays Warsaw 2015 and I was asked to replace him on very short notice. </p>
<p>Fortunately I had some slides from other events, merged
those very quickly, added some more and tried to improvise on the scene ;)</p>
<p><center>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/7enDSiQ4NYiYkI" width="510" height="420" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/d0cent/containers-do-not-contain" title=""Containers do not contain"" target="_blank">"Containers do not contain"</a> </strong> from <strong><a href="//www.slideshare.net/d0cent" target="_blank">Maciej Lasyk</a></strong> </div>
</center></p>Autumn conferences recap2015-11-26T00:00:00+01:002015-11-26T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-11-26:2015/Nov/26/autumn-conferences-recap/<p>I do like taking part in conferences. I still think, that conferences are one of the most important places where people
get motivated (e.g. by meeting influencers, talking to other engineers, listening to completely new ideas or having
opportunity to ask).</p>
<p>Also I'm certain that thanks to conferences we improve ourselves. We improve our communication & soft skills, we learn
to ask proper questions or polish our engineering skills during deep technical discussions. And also we actually have
the opportunity to show others our achievements during ignite talks.</p>
<p>Also I really do love travelling to new places. My private motivation behind this is always the same - to have even a
while for a short run in a new place; just like this:</p>
<p><center><img alt="Lazienki" src="https://maciej.lasyk.info/images/lazienki.jpg" /></center></p>
<p>Last but not least. Always try to commute using some public bicycle rentals. Thanks to that we have this unique
opportunity to actually see the those interesting parts of cities.</p>
<h2>systemd.event</h2>
<p><center><img alt="systemd" src="https://maciej.lasyk.info/images/systemd-event-2015-lennart.jpg" /></center></p>
<p>This was the first edition of <strong>systemd</strong> conference. It took place in Berlin, place where
<a href="https://en.wikipedia.org/wiki/Lennart_Poettering">Lennart Poettering</a> lies in his cave ;)</p>
<p>I really looked forward this event, so many thanks to <a href="https://en.wikipedia.org/wiki/Lennart_Poettering">Marcin Skarbek</a>
for the idea of travelling to Berlin.</p>
<p>What I liked about the event?</p>
<ul>
<li><strong>agenda</strong> - basically most of presentations were deeply, technical about systemd; bullshit factor: 1/10 - that's a
great score!</li>
<li><strong>hackfest</strong> - during 3rd day we took part in a hackfest. we wrote down some ideas and worked on those. My idea was
not that strictly connected to systemd; I just wanted to run some service (GoCD) on different containerization engines
so I'd be able to compare those and make my own opinion. I've put everything I did there on
<a href="https://github.com/docent-net/systemd-conference-2015-hacking">github</a> - not that much as I was struggling with
some Docker/FS issue for a majority of time (generally speaking <strong>docker-storage-setup</strong> is an abomination)</li>
<li><strong>venue</strong> - conference was placed in a co-working place with 7/10 hipster factor ;) The room was not that huge - but
there were no problems with crowd or whatever - there was plenty of space</li>
<li><strong>city</strong> - I've been to Berlin for the first time and I must say - I liked it!
<a href="https://plus.google.com/+MaciejLasykDocent/posts/LsgcAWfeR4H">I've put some photos in G+</a></li>
</ul>
<p><strong>Which talks I liked the most?</strong></p>
<ul>
<li><a href="https://www.youtube.com/watch?v=9Sq6FgOW6p8">Fedora Chicanery: How to use systemd nonsense hacks to solve real problems</a>
by <a href="https://twitter.com/sgallagh_redhat">Stephen Gallagher</a></li>
<li><a href="https://www.youtube.com/watch?v=eqr_rtlNY_Y">journald and the new options for remote logging</a> by
Zbigniew Jędrzejewski-Szmek</li>
<li><a href="https://www.youtube.com/watch?v=zVy1lQqJJj8">Making Docker and systemd play nice together</a> by <a href="https://twitter.com/rhatdan">Dan Walsh</a></li>
<li><a href="https://www.youtube.com/watch?v=wVk-NWtiIZY">Challenges Deploying Over 5,000 Containers per Host</a> by
<a href="https://twitter.com/DavidStrauss">David Strauss</a></li>
<li><a href="https://www.youtube.com/watch?v=7CWmuhkgZWs">systemd and Control Groups</a> by Lennart Poettering</li>
</ul>
<p>The official webpage of conference is here: <a href="https://systemd.events/">https://systemd.events/</a>. You can also watch
all the <a href="https://www.youtube.com/channel/UCvq_RgZp3kljp9X8Io9Z1DA">videos here on youtube channel</a>.</p>
<p>Of course remember to <a href="https://wiki.freedesktop.org/www/Software/systemd/">check the official systemd page</a></p>
<h2>pycon.cz</h2>
<p><center><img alt="pycon" src="https://maciej.lasyk.info/images/pycon-cz-logo.png" /></center></p>
<p>Actually this was first conference where I saw serving vodka with Redbulls (starting from.. 11a.m.?). Incredible :D</p>
<p>About the conference - this was my first Python conference, and just like with the systemd.event I really liked the idea
of one day of talks and second day of workshops and sprints.</p>
<p>Also just couldn't resist and not take a photo on the Fedora booth ;)</p>
<p><center><img alt="fedorabooth" src="https://maciej.lasyk.info/images/pycon-cz-fedora.jpg" /></center></p>
<p><strong>About presentations I liked:</strong></p>
<ul>
<li><a href="https://speakerdeck.com/hawkowl/the-future-of-twisted-and-pretty-much-everything-else-pycon-cz-keynote-2015">The future of Twisted</a>
by <a href="https://twitter.com/hawkieowl">Amber Brown</a></li>
<li><strong>So you have an Python app and now what?</strong> by <a href="https://twitter.com/verosk">Věroš Kaplan</a></li>
</ul>
<p>On the second day there were workshops / sprint sessions. I took part in webscraping workshop and GIS. Everything I
worked on during those events is now placed <a href="https://github.com/docent-net/pyconcz-2015-hacking">on my github account</a></p>
<p>I personally traveled those 300km mainly to take part in workshops and I must say that i really liked it! I found it
very helpful and well organized. Also the workshop venue was great - yet another co-working space with hipster factor
8/10 (even more than in Berlin in systemd.conf!)</p>
<p>Many thanks to organizers - keep it this way guys!</p>
<h2>Devopsdays Warsaw 2015</h2>
<p><center><img alt="Devopsdays" src="https://maciej.lasyk.info/images/devopsdays-warsaw-logo.png" /></center></p>
<p>This year was a second edition of <strong>Devopsdays</strong> here in Poland. We hosted many great
<a href="http://2015.devopsdays.pl/agenda/speakers/">presenters</a> and audience.</p>
<p>This time I was asked to be part of the Program Committee (thanks for that guys!), so I had this unique opportunity
to actually have an impact on the agenda of this event. We all did our best to make sure that this would be an awesome
conference :)</p>
<p>Actually there were many good ideas (Ignite Talks and Open Spaces were the most interesting imo).</p>
<p><strong>Which presentation I liked the most?</strong></p>
<ul>
<li><a href="http://2015.devopsdays.pl/agenda/talk/the-dark-art-of-container-monitoring/">The dark art of container monitoring</a> by
<a href="https://twitter.com/luca3m">Luca Marturana</a>. It's about tool that should be already well known by systems engineers - it
is called <a href="https://sysdig.com/">Sysdig</a> and it rocks in terms of OS excavations. </li>
<li><a href="http://2015.devopsdays.pl/agenda/talk/hidden-cost-microservices-architecture/">The hidden cost of the microservices architecture</a>
by <a href="https://twitter.com/miroslawnagas">Mirosław Nagaś</a>. Mirek told a story of impelmenting Microservices in <a href="https://lab.getbase.com/">Base</a>
company. This was a classical case-study, which are so needed on conferences. I really appreciate when people share
their experiences - many props for that Mirek!</li>
<li><a href="http://2015.devopsdays.pl/agenda/talk/jaac-jenkins-as-a-code/">Jenkins as a Code</a> by
<a href="https://twitter.com/wybczu">Łukasz Szczęsny</a>. Łukasz showed us, how to approach Jenkins in an automated way forgetting
non-fancy UI, hell of configuration-clickin' and maintaining dependencies between jobs. And he actually used Ansible -
so many props ;)</li>
</ul>
<p><strong>Ignite talks</strong></p>
<p><center><img alt="Devopsdays Openspaces" src="https://maciej.lasyk.info/images/devopsdays-openspaces.jpg" /></center></p>
<p>This was a great and simple idea: people write down subjects to discuss, then all of us vote for the most interesting
topics and finally we start discussions.</p>
<p>My favorite ones:</p>
<ul>
<li><strong>life and work balance</strong> - it was about understanding how we get the idea of work-life balance and where're limits</li>
<li><strong>Continuous Delivery - success stories</strong> - self explaining ;)</li>
<li><strong>Hard vs Soft skills</strong> - oh seriously we discussed everything that was not technical-related</li>
<li><strong>Moving to systemd</strong> - we tried to answer questions about systemd</li>
</ul>Running sshd on 443/tcp2015-11-17T00:00:00+01:002015-11-17T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-11-17:2015/Nov/17/running-sshd-on-443tcp/<p><center><img alt="Selinux" src="https://maciej.lasyk.info/images/selinux-logo.png" /></center></p>
<h3>What?</h3>
<p>So I had this need to run <strong>ssh</strong> daemon on port 443/TCP. I reconfigured sshd daemon, iptables and stucked with Selinux
policy, when I wanted to assign port 443 to sshd_port_t (so I thought I'd need to remove in the first step actual
assignment of port 443 which is http_port_t):</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@srv<span class="w"> </span>~<span class="o">]</span><span class="c1"># semanage port -d -t http_port_t -p tcp 443 </span>
ValueError:<span class="w"> </span>Port<span class="w"> </span>tcp/443<span class="w"> </span>is<span class="w"> </span>defined<span class="w"> </span><span class="k">in</span><span class="w"> </span>policy,<span class="w"> </span>cannot<span class="w"> </span>be<span class="w"> </span>deleted
</pre></div>
<h3>SSH on port 443? sshd configuration + iptables</h3>
<p>Yup, it happens ;)</p>
<p>It's simple reconfiguration done in <strong>/etc/ssh/sshd_config</strong>:</p>
<div class="highlight"><pre><span></span>Port<span class="w"> </span><span class="m">22</span>
Port<span class="w"> </span><span class="m">443</span>
</pre></div>
<p>So as you can see I left sshd running also on port 22/tcp. Also reconfigured iptables (Centos 7, no firewalld yet):</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@srv<span class="w"> </span>~<span class="o">]</span><span class="c1"># grep 443 /etc/sysconfig/iptables</span>
-A<span class="w"> </span>INPUT<span class="w"> </span>-p<span class="w"> </span>tcp<span class="w"> </span>-m<span class="w"> </span>state<span class="w"> </span>--state<span class="w"> </span>NEW<span class="w"> </span>-m<span class="w"> </span>tcp<span class="w"> </span>--dport<span class="w"> </span><span class="m">443</span><span class="w"> </span>-j<span class="w"> </span>ACCEPT
</pre></div>
<h3>Back to Selinux - how to fix this?</h3>
<p>Actually answer lays in documentation:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>man<span class="w"> </span>semanage-port
<span class="w"> </span>-m,<span class="w"> </span>--modify
<span class="w"> </span>Modify<span class="w"> </span>a<span class="w"> </span>record<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>specified<span class="w"> </span>object<span class="w"> </span><span class="nb">type</span>
</pre></div>
<p>So basically instead of removing this httpd_port_t assignment, what is impossible without recompiling the policy I just
modified it:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@srv<span class="w"> </span>~<span class="o">]</span><span class="c1"># semanage port -m -t ssh_port_t -p tcp 443</span>
</pre></div>
<p>And now we have:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@netrunner<span class="w"> </span>~<span class="o">]</span><span class="c1"># semanage port -l | grep 443 </span>
http_port_t<span class="w"> </span>tcp<span class="w"> </span><span class="m">80</span>,<span class="w"> </span><span class="m">81</span>,<span class="w"> </span><span class="m">443</span>,<span class="w"> </span><span class="m">488</span>,<span class="w"> </span><span class="m">8008</span>,<span class="w"> </span><span class="m">8009</span>,<span class="w"> </span><span class="m">8443</span>,<span class="w"> </span><span class="m">9000</span>
ssh_port_t<span class="w"> </span>tcp<span class="w"> </span><span class="m">443</span>,<span class="w"> </span><span class="m">444</span>,<span class="w"> </span><span class="m">22</span>
</pre></div>
<p>Works for me ;)</p>SELinux / sVirt vs Venom vulnerability2015-05-13T00:00:00+02:002015-05-13T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2015-05-13:2015/May/13/selinux-svirt-vs-venom-vulnerability/<p><center><img alt="libvirt" src="https://maciej.lasyk.info/images/venom.png" /></center></p>
<h1>Venom?</h1>
<p>Basically it's a vulnerability that may be used to escape hypervisor (Qemu
based, e.g. KVM. XEN) and obtain root permissions on host machine (and from
that on also other virtual machines).</p>
<p>This vulnerability has interesting PR - nice logo, name and a website
:O Real triplet of security marketing: <a href="http://venom.crowdstrike.com/">Venom
webpage</a></p>
<p>Also check <a href="https://access.redhat.com/articles/1444903">RedHat article on this
CVE</a></p>
<h1>Am I fucked?</h1>
<p>Maybe. If you're using Qemu driver then you might have a problem...</p>
<p>...but...</p>
<p>... <strong>not when using SELinux / sVirt</strong></p>
<p>With <a href="http://www.selinuxproject.org/page/SVirt">sVirt</a> and SELinux securing
your system <a href="https://securityblog.redhat.com/2015/05/13/venom-dont-get-bitten/">you should sleep
well</a>.
Because even if attacker successfully escapes hypervisor confinement he will be
still under the dome of SELinux confinement (which by default is set to
<strong>system_u:system_r:svirt_t</strong>; </p>
<p>But <strong>should</strong> may not be enough. So I'm gonna say this - you <strong>can</strong> sleep
well thanks to MCS (Multi-Level security) & MLC (Multi-Category security)
in SELinux.</p>
<p>What I wrote above tells us that every virtual machine has <strong>svirt_t</strong> security
type by default. This is a confinement configuration for each VM. And there is
more. Each virtualized guest process is labeled and runs with a dynamically
generated level. Each process is isolated from other VMs with different MCS
levels. Example from one of my boxes:</p>
<div class="highlight"><pre><span></span>system_u:system_r:svirt_t:s0:c14,c894
system_u:system_r:svirt_t:s0:c393,c652
system_u:system_r:svirt_t:s0:c730,c767
system_u:system_r:svirt_t:s0:c465,c623
system_u:system_r:svirt_t:s0:c246,c630
system_u:system_r:svirt_t:s0:c37,c395
system_u:system_r:svirt_t:s0:c225,c437
system_u:system_r:svirt_t:s0:c40,c846
system_u:system_r:svirt_t:s0:c523,c530
system_u:system_r:svirt_t:s0:c361,c630
system_u:system_r:svirt_t:s0:c63,c827
system_u:system_r:svirt_t:s0:c52,c133
system_u:system_r:svirt_t:s0:c416,c722
system_u:system_r:svirt_t:s0:c36,c814
system_u:system_r:svirt_t:s0:c615,c661
system_u:system_r:svirt_t:s0:c96,c792
system_u:system_r:svirt_t:s0:c60,c74
system_u:system_r:svirt_t:s0:c492,c659
system_u:system_r:svirt_t:s0:c72,c693
system_u:system_r:svirt_t:s0:c499,c536
</pre></div>
<p>This is just a slice. As you can see each process (VM) has other MLC.</p>
<h1>Solution?</h1>
<p>Yup - even SELinux might have some security vulnerabilities we don't know about
so it's better to fix Qemu. How? Simply:</p>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>update<span class="w"> </span>qemu-kvm
</pre></div>
<p>(or update all pkgs - whatever)</p>
<p>EDIT: <a href="http://danwalsh.livejournal.com/71489.html">Dan Walsh confirmed</a> that Venom is not an issue w/ SELinux enabled.</p>libvirt-daemon-driver-storage bug in Centos 72015-04-06T00:00:00+02:002015-04-06T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2015-04-06:2015/Apr/06/libvirt-daemon-driver-storage-bug-in-centos-7/<p><center><img alt="libvirt" src="https://maciej.lasyk.info/images/libvirt.png" /></center></p>
<h1>failed to load module libvirt_driver_storage.so</h1>
<p>Today I was installing
<a href="http://libguestfs.org/virt-sysprep.1.html">virt-sysprep</a> tool on one of KVM
hosts. When this was done I couldn't list VMs anymore using <strong>virsh</strong>:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@host1<span class="w"> </span>~<span class="o">]</span><span class="c1"># virsh list --all</span>
<span class="o">[</span><span class="m">454</span>/564<span class="o">]</span>
<span class="w"> </span>Id<span class="w"> </span>Name<span class="w"> </span>State
<span class="w"> </span>----------------------------------------------------
</pre></div>
<p>Hmm that's not normal obsiously. Checked the <strong>libvirtd</strong> status:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@host1<span class="w"> </span>~<span class="o">]</span><span class="c1"># service libvirtd status</span>
Redirecting<span class="w"> </span>to<span class="w"> </span>/bin/systemctl<span class="w"> </span>status<span class="w"> </span>libvirtd.service
libvirtd.service<span class="w"> </span>-<span class="w"> </span>Virtualization<span class="w"> </span>daemon
<span class="w"> </span>Loaded:<span class="w"> </span>loaded<span class="w"> </span><span class="o">(</span>/usr/lib/systemd/system/libvirtd.service<span class="p">;</span><span class="w"> </span>enabled<span class="o">)</span>
<span class="w"> </span>Active:<span class="w"> </span>active<span class="w"> </span><span class="o">(</span>running<span class="o">)</span><span class="w"> </span>since<span class="w"> </span>Mon<span class="w"> </span><span class="m">2015</span>-04-06<span class="w"> </span><span class="m">15</span>:43:55<span class="w"> </span>CEST<span class="p">;</span><span class="w"> </span>1h<span class="w"> </span>16min<span class="w"> </span>ago
<span class="w"> </span>Docs:<span class="w"> </span>man:libvirtd<span class="o">(</span><span class="m">8</span><span class="o">)</span>
<span class="w"> </span>http://libvirt.org
<span class="w"> </span>Main<span class="w"> </span>PID:<span class="w"> </span><span class="m">21191</span><span class="w"> </span><span class="o">(</span>libvirtd<span class="o">)</span>
<span class="w"> </span>CGroup:<span class="w"> </span>/system.slice/libvirtd.service
<span class="w"> </span>├─<span class="w"> </span><span class="m">5865</span><span class="w"> </span>/sbin/dnsmasq<span class="w"> </span>--conf-file<span class="o">=</span>/var/lib/libvirt/dnsmasq/default.conf
<span class="w"> </span>└─21191<span class="w"> </span>/usr/sbin/libvirtd
...
Apr<span class="w"> </span><span class="m">06</span><span class="w"> </span><span class="m">17</span>:01:42<span class="w"> </span>host1.devops.ent<span class="w"> </span>libvirtd<span class="o">[</span><span class="m">6254</span><span class="o">]</span>:<span class="w"> </span>failed<span class="w"> </span>to<span class="w"> </span>load<span class="w"> </span>module<span class="w"> </span>
/usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so<span class="w"> </span>
/usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so:<span class="w"> </span>symbol<span class="w"> </span>
dm_task_get_info_with_deferred_remove,<span class="w"> </span>version<span class="w"> </span>Base<span class="w"> </span>not<span class="w"> </span>defined<span class="w"> </span><span class="k">in</span><span class="w"> </span>file<span class="w"> </span>
libdevmapper.so.1.02<span class="w"> </span>with<span class="w"> </span>link<span class="w"> </span><span class="nb">time</span><span class="w"> </span>reference
Apr<span class="w"> </span><span class="m">06</span><span class="w"> </span><span class="m">17</span>:01:42<span class="w"> </span>host1.devops.ent<span class="w"> </span>libvirtd<span class="o">[</span><span class="m">6254</span><span class="o">]</span>:<span class="w"> </span>failed<span class="w"> </span>to<span class="w"> </span>load<span class="w"> </span>module<span class="w"> </span>
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so<span class="w"> </span>
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so:<span class="w"> </span>undefined<span class="w"> </span>
symbol:<span class="w"> </span>virStorageFileCreate
...
</pre></div>
<p>I confirmed that installing <strong>virt-sysprep</strong> caused also upgrading libvirtd related packages:</p>
<div class="highlight"><pre><span></span><span class="w"> </span>Updated<span class="w"> </span>libvirt-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-client-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-config-network-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-config-nwfilter-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-driver-interface-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-driver-lxc-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-driver-network-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-driver-nodedev-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-driver-nwfilter-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-driver-qemu-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-driver-secret-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Updated<span class="w"> </span>libvirt-daemon-driver-storage-1.1.1-29.el7_0.7.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Update<span class="w"> </span><span class="m">1</span>.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
<span class="w"> </span>Dep-Install<span class="w"> </span>libvirt-daemon-kvm-1.2.8-16.el7_1.2.x86_64<span class="w"> </span>@updates
</pre></div>
<p>So once more I read the problematic log entry:</p>
<div class="highlight"><pre><span></span>failed<span class="w"> </span>to<span class="w"> </span>load<span class="w"> </span>module<span class="w"> </span>/usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so<span class="w"> </span>/usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so:<span class="w"> </span>symbol<span class="w"> </span>dm_task_get_info_wit
h_deferred_remove,<span class="w"> </span>version<span class="w"> </span>Base<span class="w"> </span>not<span class="w"> </span>defined<span class="w"> </span><span class="k">in</span><span class="w"> </span>file<span class="w"> </span>libdevmapper.so.1.02<span class="w"> </span>with<span class="w"> </span>link<span class="w"> </span><span class="nb">time</span><span class="w"> </span>reference
</pre></div>
<p>and decided to <strong>yum upgrade device-mapper-libs</strong>. Afterwards I restarted <strong>libvirtd</strong>
and everything started working perfectly.</p>
<p>This would not happen if I had updated all the libraries, but unfortunately
all I wanted was <strong>virt-sysprep</strong> installed - not update'ing all packages.</p>
<p>Finally <a href="http://bugs.centos.org/view.php?id=8403">filed a bugrequest</a></p>About cultural change w/Devops2015-03-26T00:00:00+01:002015-03-26T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-03-26:2015/Mar/26/about-cultural-change-wdevops/<p><center><img alt="Atmosphere Shuttle" src="https://maciej.lasyk.info/images/atmosphere-1.jpg" /></center></p>
<p>This time without any comment - just slides from presentation I gave today in
<a href="http://www.meetup.com/Atmosphere/events/221099327/">Atmosphere Shuttle #01</a>
meetup:</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/46335851" width="425"
height="355" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"
style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width:
100%;" allowfullscreen> </iframe>
<p><div style="margin-bottom:5px"> <strong> <a
href="//www.slideshare.net/d0cent/about-cultural-change-wdevops" title="About
cultural change w/Devops" target="_blank">About cultural change w/Devops</a>
</strong> from <strong><a href="//www.slideshare.net/d0cent"
target="_blank">Maciej Lasyk</a></strong> </div></p>Cheap SSD arrays for Continuous Integration2015-03-14T00:00:00+01:002015-03-14T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-03-14:2015/Mar/14/cheap-ssd-arrays-for-continuous-integration/<p><center><img alt="Its me!" src="https://maciej.lasyk.info/images/ssd_array.jpg" /></center></p>
<h3>The need for iop/s</h3>
<p>I think that every sysdamin has this need. It is like "I have a need - the need
for speed". So basically when I see 400 operations per second as upper limit on
some SATA drives the I cry very loudly.</p>
<p>But bussiness can't always provide RAID10 array of 4x240GB Intel SSD drives /
$4000.</p>
<p>I faced this problem lately. While improving environment performance for continuous integration I noticed how drastically we have this need for iop/s on every Jenkins and Selenium nodes.</p>
<p>So we have this bleeding-edge-storage which has couple of enterprise class SSDs
but honestly I didn't want to sacrifice those holy part of our infrastructure
for some ephemeral work.</p>
<p>I just wanted some drives which would provide me w/about 15-20k iop/s. Simple,
cheap SSD drives.</p>
<p>So I found, that our servers actually have a free drive slots inside. Unused.
Just waiting to be fulfilled with something. Think local storage :D</p>
<p>And this was it. What about buying desktop - class SSD drives?</p>
<h3>Trim issue?</h3>
<p>It is not an issue. Just make sure you've got <a href="https://turriebuntu.wordpress.com/ubuntu-pages/precise-specific-pages/using-fstrim-to-trim-your-ssd-instead-of-delete-in-fstab/">fstrim
running</a>
and remember to <a href="http://blog.neutrino.es/2013/howto-properly-activate-trim-for-your-ssd-on-linux-fstrim-lvm-and-dmcrypt/">trim on all filesystem
layers(!)</a></p>
<h3>So... what kind of drives?</h3>
<p><a href="http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead">According to
this</a>
The best price to longevity ratio will give you Kingston HyperX drives ($100
each 240GB drive). So basically you can set up RAID1 for $200 or RAID10 for
$400.</p>
<p>Of course this is just for some temporary data. Those kind of arrays should't
be there for crucial datastores. And of course - remember about capacity
planning. When I see 600TB written as the moment in time when the drive will be
dead I can basically say, that $200 RAID array will need to be replaced in 18
months. Wow - this is really nice lifetime for this at this price. So I actualy
created 2xRAID10 arrays (240GB Kingston HyperX drives) what gave me 2x0,5TB in
RAID10 / local storage for $1000. And I'm gonna replace this in about 1,5 year
(maybe still as a subject of warranty).</p>
<p>Great deal imo. If I did want to but enterprise class disks (like Intel) for
this purpose I would have to pay 8*$1000 and that is a horrible amount of
goldies.</p>
<p>Summing this up - CI (Jenkins, Selenium, others) + local storage on cheap SSD drives = <3</p>Conferences schedule for Spring 20152015-03-12T00:00:00+01:002015-03-12T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-03-12:2015/Mar/12/conferences-schedule-for-spring-2015/<p><center><img alt="Its me!" src="https://maciej.lasyk.info/images/maciek.jpg" /></center></p>
<h3>Spring is near, so conference season is almost here!</h3>
<p>As a friend of mine <a href="https://twitter.com/j_palka">Jarek Pałka</a> told me
yesterday - this is like a snowball. You give your first talk, then next and
next and you event don't blink an eye and you find yourself in a place where
some year you have 5 talks scheduled in a row during one month.</p>
<p>But.. I really do like it. Conferences are great. Just like a sport - you
prepare, train, test, rest and again over and over again. Then there is this
day where you can finally unleash accumulated endorphines and let yourself into
this flow of talks, discussions, chip-chats and off-the-record confabulations
;)</p>
<p>And then there is it. Couple of weeks later someone tries to get hired in your
company just because your presentation showed guys that it is really a good
place. Months later it appears that your idea was developed more and someone
else in some other event talks about it. There is more - you can attract people
to any Open-Source project you want - just tell them why it rocks!</p>
<p>Also it is the only place when you can actually meet the fundaments of our IT
world. One of my biggest dreams was photo w/Linus Torvalds. And here it is
(taken 3 years ago at LinuxCon Barcelona):</p>
<p><center><img alt="LinuxCon" src="https://maciej.lasyk.info/images/linuxcon.jpeg" /></center></p>
<p>Ahh I simply <3 that. Also I do love travelling w/my pair of running shoes
getting up @6:00 just to have a morning portion of endorphines before starting
next conference day on a route I completely don't know.</p>
<h3>The Spring Schedule aka "Come get some!"</h3>
<ul>
<li><a href="http://opensourcesummit.pl">OpenSource Summit 2015</a> - 2015-03-24 - this is a
one - day event, which will bring together all the greatest minds of our
Polish Open - source scene. I'll give there talk about <a href="http://opensourcesummit.pl/maciej-lasyk/">Docker
containers</a>. And finally I'll have
opportunity to see <a href="">Grzegorz Nosek's</a> presentation about Ansible and take
part in his trainings!</li>
<li>#DevOps meeting, Atmosphere Shuttle - 2015-03-26 - this is a DevOps meetup
where I'll give a talk about changing old & stiff envs into more devops. This
event is part of a tour which main job is to introduce Atmosphere Conference.</li>
<li><a href="http://12.sesja.linuksowa.pl/">12th Linux Session</a> - 2015-04-(18-19) - I'll
give here a talk about Orchestrating Docker Containers. And there will be
Lennart Poettering speaking there, so fingers crossed to catch him and thanks
him for his great work on Systemd!</li>
<li><a href="http://www.4developers.org.pl/">4Developers</a> - 2015-04-20 - I'll have 2
talks here. First in the "Application Architectures" track titled [Under the
dome (of failure driven
pipeline)](http://www.4developers.org.pl/en/agenda/lecture/under-dome-failure-driven-pipeline/]
and second <strong>Continuous Security in DevOps</strong> (which is still not published) in
the security track.</li>
<li><a href="http://www.atmosphere-conference.com/">Atmosphere Conference</a> -
2015-05-(18-19) - this is the biggest DevOps event in Poland. I'll be there
for the second time giving a talk about Enterprise DevOps.</li>
</ul>Dropbox and security of your files in Linux2015-03-03T00:00:00+01:002015-03-03T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-03-03:2015/Mar/03/dropbox-and-security-of-your-files-in-linux/<p><center><img alt="RHEL" src="https://maciej.lasyk.info/images/dropbox-logo.png" /></center></p>
<h3>WAT?</h3>
<p>So there's been discussion about <a href="http://www.e-siber.com/guvenlik/dropbox-accesses-all-the-files-in-your-pc-not-just-sync-folder-and-steals-everything/">Dropbox indexing all files on Windows
boxes</a></p>
<p>It basically tells that Dropbox scans all the files in the whole drive:</p>
<blockquote>
<p>Dropbox syncs not only its own folder but also everything in local drive (C:)
without any user consent or permission.</p>
</blockquote>
<p><center><img alt="WAT" src="https://maciej.lasyk.info/images/wat.jpg" /></center></p>
<h3>How about Linux?</h3>
<p>First of all - if you're sane Linux user than you use one of the <a href="https://en.wikipedia.org/wiki/Linux_Security_Modules">Linux
Security Modules (LSMs)</a>
and literally don't give a fuck. Even when <a href="http://www.techrepublic.com/article/moving-steams-local-folder-deletes-all-user-files-on-linux/">Steam wants to delete all your files
in
hard-drive</a>
you show him a middle finger and praise <a href="http://danwalsh.livejournal.com/">Dan Walsh for his
work</a></p>
<p>I'll tell this one more time - use SELinux. <a href="https://www.packtpub.com/networking-and-servers/selinux-system-administration">Read this
book</a>
- it's only 100 pages for the sake. Or use AppArmor if SELinux is too hard for
you. Then you will never find yourself again in such hard position.</p>
<h3>Erm.. Linux again?</h3>
<p>Ok, let's say you don't use SELinux (because if you did you wouldn't have to
read this).</p>
<p>So I run Dropbox (default datadir in ~/Dropbox) and fetched its PID and attached strace:</p>
<div class="highlight"><pre><span></span>strace<span class="w"> </span>-f<span class="w"> </span>-e<span class="w"> </span><span class="nv">trace</span><span class="o">=</span>file<span class="w"> </span>-s<span class="w"> </span><span class="m">1024</span><span class="w"> </span>-p<span class="w"> </span><span class="m">6033</span><span class="w"> </span>-o<span class="w"> </span>~/tmp/dropbox-wtf.txt
</pre></div>
<p>Now I tried to create a file outside of ~/Dropbox directory:</p>
<div class="highlight"><pre><span></span>touch<span class="w"> </span>~/tmp/wtf.txt
</pre></div>
<p>... and ...</p>
<p><strong>N O T H I N G</strong></p>
<p><strong>Quiet</strong></p>
<p><strong>Peace</strong></p>
<p>Later on just to confirm I created a file in ~/Dropbox/tmp and got a ton of
crap from the strace - filtered out the ENOENT so it's more clear (nothing
interesting there):</p>
<div class="highlight"><pre><span></span><span class="m">6308</span><span class="w"> </span>lstat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0664</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">4</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6308</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOFOLLOW<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6308</span><span class="w"> </span>lstat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0664</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">4</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6308</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOFOLLOW<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>lstat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0664</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">4</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOFOLLOW<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">48</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>lstat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0664</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">4</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOFOLLOW<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>lstat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0664</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">4</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOFOLLOW<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>lstat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0664</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">4</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOFOLLOW<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt/.dropbox.attr"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>-1<span class="w"> </span>ENOTDIR<span class="w"> </span><span class="o">(</span>Not<span class="w"> </span>a<span class="w"> </span>directory<span class="o">)</span>
<span class="m">6294</span><span class="w"> </span>lstat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0664</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">4</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOFOLLOW<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/Dropbox/tmp/wtf.txt"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_NOCTTY<span class="p">|</span>O_NONBLOCK<span class="p">|</span>O_NOATIME<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/filecache.dbx"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0644</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">26258432</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/filecache.dbx-journal"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">48</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/var/tmp"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFDIR<span class="p">|</span>S_ISVTX<span class="p">|</span><span class="m">0777</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">4096</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>access<span class="o">(</span><span class="s2">"/var/tmp"</span>,<span class="w"> </span>R_OK<span class="p">|</span>W_OK<span class="p">|</span>X_OK<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/var/tmp/etilqs_1GCo6ezszP3OnQ1"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_EXCL<span class="p">|</span>O_NOFOLLOW<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0600</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">52</span>
<span class="m">6294</span><span class="w"> </span>unlink<span class="o">(</span><span class="s2">"/var/tmp/etilqs_1GCo6ezszP3OnQ1"</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1"</span>,<span class="w"> </span>O_RDONLY<span class="p">|</span>O_CLOEXEC<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">54</span>
<span class="m">6294</span><span class="w"> </span>unlink<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/filecache.dbx-journal"</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">35</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/dev/urandom"</span>,<span class="w"> </span>O_RDONLY<span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>open<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span>O_RDWR<span class="p">|</span>O_CREAT<span class="p">|</span>O_CLOEXEC,<span class="w"> </span><span class="m">0644</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">47</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/UPDATED_jt6wdD"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">7168</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
<span class="m">6294</span><span class="w"> </span>stat<span class="o">(</span><span class="s2">"/home/putinstopattackingukraine/.dropbox/instance1/PENDING_FIbd69"</span>,<span class="w"> </span><span class="o">{</span><span class="nv">st_mode</span><span class="o">=</span>S_IFREG<span class="p">|</span><span class="m">0600</span>,<span class="w"> </span><span class="nv">st_size</span><span class="o">=</span><span class="m">5120</span>,<span class="w"> </span>...<span class="o">})</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
</pre></div>
<p>So we basically see that Dropbox reads only its datadir (~/Dropbox), config dir
(~/.dropbox), urandom device (probably generating bitcoins, but this is ok),
and /var/tmp (not sure however what kind of tmpfiles it creates - but this
would be rather performance question).</p>
<p>So we're safe. Dropbox does not read all the file contents on Linux.</p>
<p><strong>But once again - start using Linux Security Modules!</strong></p>Dracut, netboot and default gateway in RHEL/CentOS 72015-02-18T00:00:00+01:002015-02-18T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-02-18:2015/Feb/18/dracut-netboot-and-default-gateway-in-rhelcentos-7/<p><center><img alt="RHEL" src="https://maciej.lasyk.info/images/rhel-logo-small.png" /></center></p>
<h3>WAT?</h3>
<p>Ok, so I've rolled up a new host and made it boot OS from the iSCSI drive.
Great. I have 4 NICs in this box.</p>
<p>After booting machine default gateway was set up on the iSCSI network. Weird,
but ok - I tried to change that with:</p>
<ul>
<li>Setting up a gateway in <strong>/etc/sysconfig/network</strong> by entering <strong>GATEWAY</strong>
param; didn't work, still defroute on iSCSI NIC</li>
<li>Setting up <strong>DEFROUTE="no"</strong> in /etc/sysconfig/network-scripts/ifcfg-eno2
(the iSCSI NIC). Didn't work, still defroute on iSCSI NIC</li>
<li>Use "nmcli" to setup <strong>ipv4.never-default</strong> for the iSCSI NIC. Didn't work -
this only rewrites ifcfg* script adding DEFROUTE param</li>
</ul>
<h3>So What?</h3>
<p>Yup, things were going weirdo and honestly I left this for a while as I simply
didn't find the answer - even after debugging the
/etc/sysconfig/network-scripts/* scripts. Something - not NetworkManager - was
overwriting the default route during boot after NetworkManager did it's job.</p>
<p>I came back to this problem after while and poked around this one ifcfg config
for the iSCSI NIC:</p>
<div class="highlight"><pre><span></span><span class="c1"># Generated by dracut initrd</span>
<span class="nv">DEVICE</span><span class="o">=</span><span class="s2">"eno2"</span>
<span class="nv">ONBOOT</span><span class="o">=</span>yes
<span class="nv">NETBOOT</span><span class="o">=</span>yes
<span class="nv">UUID</span><span class="o">=</span><span class="s2">"whatevercdff28f030fd"</span>
<span class="nv">BOOTPROTO</span><span class="o">=</span>none
<span class="nv">IPADDR</span><span class="o">=</span><span class="s2">"192.168.1.2"</span>
<span class="nv">GATEWAY</span><span class="o">=</span><span class="s2">"192.168.1.1"</span>
<span class="nv">NETMASK</span><span class="o">=</span><span class="s2">"255.255.255.0"</span>
<span class="nv">HWADDR</span><span class="o">=</span><span class="s2">"00:22:10:d5:a2:6d"</span>
<span class="nv">TYPE</span><span class="o">=</span>Ethernet
<span class="nv">NAME</span><span class="o">=</span><span class="s2">"eno2"</span>
</pre></div>
<p>And finally I payed attention to this comment: <strong>Generated by dracut initrd</strong>.
While installing this host I had to reconfigure dracut a little bit to make
grub boot this OS from network via iSCSI (I wrote about <a href="https://maciej.lasyk.info/2015/Feb/09/iscsi-boot-in-rhelcentos7-broken/">it here</a>
as this is a bug in CentOS/RHEL7). In <strong>/etc/default/grub</strong> I entered:</p>
<div class="highlight"><pre><span></span><span class="nv">GRUB_TIMEOUT</span><span class="o">=</span><span class="m">5</span>
<span class="nv">GRUB_DISTRIBUTOR</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>sed<span class="w"> </span><span class="s1">'s, release .*$,,g'</span><span class="w"> </span>/etc/system-release<span class="k">)</span><span class="s2">"</span>
<span class="nv">GRUB_DEFAULT</span><span class="o">=</span>saved
<span class="nv">GRUB_DISABLE_SUBMENU</span><span class="o">=</span><span class="nb">true</span>
<span class="nv">GRUB_TERMINAL_OUTPUT</span><span class="o">=</span><span class="s2">"console"</span>
<span class="nv">GRUB_CMDLINE_LINUX</span><span class="o">=</span><span class="s2">"ip=192.168.1.2::192.168.1.1:255.255.255.0:[...]</span>
<span class="s2">GRUB_DISABLE_RECOVERY="</span>true<span class="s2">"</span>
</pre></div>
<p>That's it! This iSCSI network is a simple P2P connection, so it does not need
the gateway there! Basing on <a href="http://people.redhat.com/harald/dracut.html#id536255">this article</a>
the proper format for dracut is:</p>
<div class="highlight"><pre><span></span><span class="nv">ip</span><span class="o">=</span><client-IP>:<span class="o">[</span><span class="w"> </span><server-id><span class="w"> </span><span class="o">]</span>:<gateway-IP>:
<span class="w"> </span><netmask>:<client_hostname>:<interface>:<span class="o">{</span>none<span class="p">|</span>off<span class="o">}</span>
</pre></div>
<p>So I simply removed the gateway <strong>192.168.1.1</strong> from the grub default config,
regenerated grub with:</p>
<div class="highlight"><pre><span></span>grub2-mkconfig<span class="w"> </span>-o<span class="w"> </span>/boot/grub2/grub.cfg
</pre></div>
<p>And after reboot my default gateway was set up exactly like I configured it in
<strong>/etc/sysconfig/network</strong></p>
<p>Yes - that was truly pain in the ass!</p>iSCSI boot in RHEL/CentOS7 broken2015-02-09T00:00:00+01:002015-02-09T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-02-09:2015/Feb/09/iscsi-boot-in-rhelcentos7-broken/<p><center><img alt="RHEL" src="https://maciej.lasyk.info/images/rhel-logo-small.png" /></center></p>
<h3>WAT?</h3>
<p>Today while rolling up new host in my company I hit the following issue after
successfully installing CentOS7 on remote iSCSI drive. I just rebooted the
machine and after GRUB done it's job I just got:</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="w"> </span><span class="m">2</span>.375332<span class="o">]</span><span class="w"> </span>i8042:<span class="w"> </span>No<span class="w"> </span>controller<span class="w"> </span>found
<span class="o">[</span><span class="w"> </span><span class="m">2</span>.592960<span class="o">]</span><span class="w"> </span>dracut:<span class="w"> </span>FATAL:<span class="w"> </span>For<span class="w"> </span>argument<span class="w"> </span>
<span class="w"> </span><span class="s1">'ip=::::localhost.localdomain:eno1: none'</span><span class="se">\n</span>Value<span class="w"> </span><span class="s1">'none'</span><span class="w"> </span>
<span class="w"> </span>without<span class="w"> </span>static<span class="w"> </span>configuration<span class="w"> </span>does<span class="w"> </span>not<span class="w"> </span>make<span class="w"> </span>sanse
<span class="o">[</span><span class="w"> </span><span class="m">2</span>.593074<span class="o">]</span><span class="w"> </span>dracut:<span class="w"> </span>Refusing<span class="w"> </span>to<span class="w"> </span><span class="k">continue</span>
<span class="o">[</span><span class="w"> </span><span class="m">3</span>.197719<span class="o">]</span><span class="w"> </span>System<span class="w"> </span>halted.
</pre></div>
<h3>What's that?</h3>
<p>The answer is <a href="https://access.redhat.com/solutions/905003">here</a>. Simply
installer is broken and it creates broken grub entry for iSCSI nodes.</p>
<h3>How to fix this?</h3>
<p>There're many ways. I just rebooted the box, entered the grub edit mode and
fixed the boot param replacing:</p>
<div class="highlight"><pre><span></span>linux16<span class="w"> </span>/vmlinuz-3.10.0-123.el7.x86_64<span class="w"> </span><span class="o">[</span>...<span class="o">]</span><span class="w"> </span>
<span class="w"> </span>vconsole.font<span class="o">=</span>latarcyrheb-sun16<span class="w"> </span>
<span class="w"> </span><span class="nv">ip</span><span class="o">=</span>::::localhost.localdomain:eno1:none<span class="w"> </span><span class="o">[</span>...<span class="o">]</span>
</pre></div>
<p>with this:</p>
<div class="highlight"><pre><span></span>linux16<span class="w"> </span>/vmlinuz-3.10.0-123.el7.x86_64<span class="w"> </span><span class="o">[</span>...<span class="o">]</span><span class="w"> </span>
<span class="w"> </span>vconsole.font<span class="o">=</span>latarcyrheb-sun16<span class="w"> </span>
<span class="w"> </span><span class="nv">ip</span><span class="o">=</span><span class="m">192</span>.168.1.15::192.168.1.1:255.255.252.0:localhost.localdomain:eno1:none<span class="w"> </span><span class="o">[</span>...<span class="o">]</span>
</pre></div>
<p>Where <strong>192.168.1.15</strong> is the initiator IP addr and <strong>192.168.1.1</strong> is the
gateway IP.</p>
<p>Of course that above is just a way to successfully boot into the system. After
boot the grub2.cfg is still broken so now it's the time for a permanent
solution:</p>
<ul>
<li>Fix above again in <strong>/etc/sysconfig/grub</strong></li>
<li>Rebuild the grub configuration: <strong><em>grub2-mkconfig -o /boot/grub2/grub.cfg</em></strong></li>
<li>That's all. Just reboot and confirm it's working fine</li>
</ul>Spotify on Fedora 212015-02-08T00:00:00+01:002015-02-08T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-02-08:2015/Feb/08/spotify-on-fedora-21/<p><center><img alt="Spotify and Fedora" src="https://maciej.lasyk.info/images/fedspotify.png" /></center></p>
<h3>Spotify on Fedora?</h3>
<p>Yup, I just got bored of running Spotify on my cellphone when working on
computer. Unfortunately there is no native support for Spotify on Fedora just
like in Debian / Ubuntu (which are preferred distros by Spotify for Linux
today).</p>
<h3>So howto Spotify and Fedora?</h3>
<p>Quit simple, but a bit kinky ;)</p>
<ul>
<li><a href="http://rpmfusion.org/Configuration">Install rpmfusion repo</a></li>
<li>Install Spotify client: <strong><em>yum install lpf-spotify-client</em></strong></li>
<li>This client has some old dependency (libgcrypt.so.11), which is not (and
will not be) in F21. However I found it in <a href="https://copr.fedoraproject.org/coprs/red/libgcrypt.so.11/">this
COPR</a>, so simple
install <a href="https://copr.fedoraproject.org/coprs/red/libgcrypt.so.11/repo/fedora-rawhide/red-libgcrypt.so.11-fedora-rawhide.repo">this
repo</a>
and <strong><em>yum install compat-libgcrypt</em></strong></li>
<li>Now just type in terminal: <strong><em>lpf update</em></strong> and you will get a popup window
w/Spotify client updater/installer.</li>
</ul>
<p>If you have any issues installing see buildlog (you can view it from this popup
window)</p>Sending Evernote notes to Kindle2015-02-05T00:00:00+01:002015-02-05T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-02-05:2015/Feb/05/sending-evernote-notes-to-kindle/<p><center><img alt="Evernote and Kindle" src="https://maciej.lasyk.info/images/everkindle.png" /></center></p>
<h3>Evernote and Kindle?</h3>
<p>I use Kindle as my main database for any books, articles or whatever I'd like
to read. I try to push anything (pdfs, notes from meetings etc) to my Kindle
so I'm always sure that nothing stays unread (that's a part of my GTD
approach).</p>
<p>But also I use Evernote for storing all my knowledge, experiences, "to-read"'s
etc. It's basically huge database of everything / anything I'd like to
preserve.</p>
<p>Using Evernote you'll surely hit the <a href="Evernote Web clipper">https://evernote.com/webclipper/</a>.
It's a browser plugin which main job is to save particular pages in Evernote.</p>
<h3>So howto Kindle & Evernote?</h3>
<p>I searched for a way to basically send any page directly from Evernote Web
Clipper to my Kindle (for future reading). And this time <strong>I didn't want to
create my own solution</strong> (which would be probably easy to implement: connect via
<a href="Evernote API">https://dev.evernote.com/doc/</a>, fetch any new notes marked for
Kindle use (e.g. by tags), send to Kindle e-mail address).</p>
<p>So I found <a href="en2kindle">http://en2kindle.com</a>. It's very simple tool - just authorize it to
read your notes within Evernote and then authorize it to be a legit sender to
your Kindle mailbox. Now just decide wheter you'd like to have all notes from
particular notebook sent to your Kindle or rather only those notes tagged with
some particular tag.</p>
<p>And that's all!</p>
<p>Enjoy and support en2kindle - it's ony $6/year!</p>
<p>Oh one more thing for people concerning about <strong>security</strong>. It simple as that -
it's as secure as keeping data in Evernote. It's internet - deal with it.</p>1st Kraków Docker Meetup!2015-01-17T00:00:00+01:002015-01-17T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-01-17:2015/Jan/17/1st-krakow-docker-meetup/<p><center><img alt="Docker Logo" src="https://maciej.lasyk.info/images/docker-logo.png" /></center></p>
<h3>Docker?</h3>
<p>Yup, <a href="http://www.docker.com">Docker</a>. It's An open platform for distributed
applications.</p>
<h3>Should I attend?</h3>
<p>If you're developer, sysadmin, DevOPS, technical leader than <strong>yes, you should
attend</strong>.</p>
<p>Why? Because Linux Containers change the IT infrastructures and the way we
create and deliver services.</p>
<h3>When and where?</h3>
<p>All event details, place, agenda, registration etc are on the meetup webpage:
<a href="http://www.meetup.com/Docker-Krakow-Poland/events/219831894/">http://www.meetup.com/Docker-Krakow-Poland/events/219831894/</a></p>Linux Kernel Networking book2015-01-15T00:00:00+01:002015-01-15T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2015-01-15:2015/Jan/15/linux-kernel-networking-book/<p><center><img alt="Linux Kernel Networking" src="https://maciej.lasyk.info/images/linuxkernelnetworking.jpg" /></center></p>
<h3>What it is about?</h3>
<p>Aaargh I wish day could be longer than 24h. This books keeps me away
from my bed.</p>
<p>Simply speaking - if you're engineer (DevOPS / systems / whatever linux based)
this is a first position to read just after passing the CCNA exams if you want
to improve your knowledge about kernel / networking internals.</p>
<h3>Should I read it?</h3>
<p>Yes.</p>
<p>Even if you're a Linux / networking Ninja (so you can share your thoughts!)</p>
<p>And if you're not a ninja and still not sure if should read this... do you know
if packets dump saved by tcpdump will show data filtered by iptables or not?
(I love asking this question as it is a good orientation point)</p>
<p>Also - do you know if you can filter L2 packets w/iptables (e.g. on KVM host
using L2 bridge)?</p>
<h3>What you will find here?</h3>
<ul>
<li>netlink sockets</li>
<li>ICMP</li>
<li>IPv4/6</li>
<li>routing</li>
<li>neighbouring</li>
<li>netfilter</li>
<li>ipsec</li>
<li>and many, many more</li>
</ul>
<p>What's important is this book is not about Kernel development. It simply
focuses on the network stack describing concepts.</p>
<p>So happy reading and many thanks to the authors!</p>GIT: clone only one file2014-11-20T00:00:00+01:002014-11-20T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-11-20:2014/Nov/20/git-clone-only-one-file/<p><center><img alt="GIT" src="https://maciej.lasyk.info/images/git.png" /></center></p>
<h3>What?</h3>
<p>So I have a quite big bare repo somewhere and needed to get only one file from
there (of course - didn't know the exact filename). Frankly my GIT-fu is not
that kinky, so the way I did it could cause a little bit of smile on someones
mouth ;)</p>
<h3>How?</h3>
<p>A slice from my notes:</p>
<div class="highlight"><pre><span></span><span class="c1"># on a local FS:</span>
$<span class="w"> </span>git<span class="w"> </span>init<span class="w"> </span>.<span class="w"> </span><span class="c1"># create empty repo on local FS</span>
$<span class="w"> </span>git<span class="w"> </span>remote<span class="w"> </span>add<span class="w"> </span>remote_name<span class="w"> </span>path<span class="w"> </span><span class="c1"># add a remote</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># on a remote, bare repo:</span>
$<span class="w"> </span>git<span class="w"> </span>log<span class="w"> </span>-n<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="c1"># get last commits id</span>
$<span class="w"> </span>git<span class="w"> </span>show<span class="w"> </span>--name-only<span class="w"> </span>commit_id<span class="w"> </span><span class="c1"># get listing of files from the last commits</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># on a local FS:</span>
$<span class="w"> </span>git<span class="w"> </span>archive<span class="w"> </span>--remote<span class="o">=</span>remote_name<span class="w"> </span>HEAD:some/dir/somewhere<span class="w"> </span>filename<span class="w"> </span><span class="p">|</span><span class="w"> </span>tar<span class="w"> </span>xf<span class="w"> </span>-
</pre></div>
<p>So this way you'll end up with only one file scrapped from the remote repo.</p>Invitation to JavaCamp #132014-11-20T00:00:00+01:002014-11-20T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-11-20:2014/Nov/20/invitation-to-javacamp-13/<p><center><img alt="PJUG" src="https://maciej.lasyk.info/images/pjug.jpeg" /></center></p>
<p>Once again I'll have a talk about Docker cluster management. This time it will
take place in the Sabre office here in Krk and will be part of the JavaCamp #13
meeting which details are on the <a href="http://www.meetup.com/Polish-Java-User-Group/events/213331502/">meetup
webpages</a></p>
<p>So - see you there!</p>
<p>Can't make it? This talk is available on the youtube (not exactly this, but
same subject which I presented on Linux Autumn Conference):</p>
<iframe width="560" height="315"
src="//www.youtube.com/embed/Qflntb2ZdlQ?rel=0" frameborder="0"
allowfullscreen></iframe>Poor man's VIM sessions2014-11-19T00:00:00+01:002014-11-19T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-11-19:2014/Nov/19/poor-mans-vim-sessions/<p><center><img alt="VIM" src="https://maciej.lasyk.info/images/vim.jpg" /></center></p>
<p>Some of guys use VIM just like Eclipse - one IDE to rule them all. I don't like
this approach - instead of this I rather work in couple of
<a href="http://tmux.sourceforge.net/">Tmux</a> windows having exactly one VIM in each (usually top - left pane). </p>
<p>If you don't know what Tmux is and how it works check this <a href="http://robots.thoughtbot.com/a-tmux-crash-course">crash course</a> or <a href="http://www.techrepublic.com/blog/linux-and-open-source/is-tmux-the-gnu-screen-killer/">this comparison between GNU Screen and Tmux</a>.</p>
<p>So - instead of opening new VIM windows and sessions manually everytime I
start my work I just start vim using bash function (placed in my zsh_profile):</p>
<blockquote>
<p>$ vims session_name</p>
</blockquote>
<p>This way I keep one session per task (just like I do with Tmux windows - think
about those like feature branches in git!).</p>
<p>So the whole workflow looks like:</p>
<ol>
<li>Create new tmux windows with my custom layout (using <a href="https://github.com/tmuxinator/tmuxinator">Tmuxinator</a>)</li>
<li>Create feature branch in git repo</li>
<li>Open VIM in the top - left pane of VIM using session named exactly the same
like feature branch</li>
</ol>
<p>Yes - I could possibly remove the last step and include it into the tmux
window creation, but remember - this is <strong>poor man's way</strong> ;)</p>
<p>And one more thing - I use <a href="https://github.com/xolox/vim-session">vim-session</a>
plugin to handle sessions in VIM.</p>
<p>And the function code? See below:</p>
<div class="highlight"><pre><span></span>vims<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nv">_vim_sess_dir</span><span class="o">=</span>~/.vim/sessions/
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="nv">$1</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"You must enter session name!"</span>
<span class="w"> </span>ls<span class="w"> </span><span class="nv">$_vim_sess_dir</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nv">_vim_sess</span><span class="o">=</span><span class="nv">$_vim_sess_dir</span>/<span class="nv">$1</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-f<span class="w"> </span><span class="nv">$_vim_sess</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>cp<span class="w"> </span><span class="nv">$_vim_sess_dir</span>/default.vim<span class="w"> </span><span class="nv">$_vim_sess</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span>vim<span class="w"> </span>-S<span class="w"> </span><span class="nv">$_vim_sess</span>
<span class="w"> </span><span class="k">fi</span>
<span class="o">}</span>
</pre></div>Tmux: join-pane2014-11-19T00:00:00+01:002014-11-19T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-11-19:2014/Nov/19/tmux-join-pane/<p><center><img alt="TMUX" src="https://maciej.lasyk.info/images/tmux.png" /></center></p>
<p>Today I finally learnt how to move a pane between windows in
<a href="http://tmux.sourceforge.net/">Tmux</a> and yes - I will share this vodoo
knowledge with you guys.</p>
<p>Searching through the internet gives us this:</p>
<div class="highlight"><pre><span></span><span class="c1"># pane movement</span>
bind-key<span class="w"> </span>j<span class="w"> </span>command-prompt<span class="w"> </span>-p<span class="w"> </span><span class="s2">"join pane from:"</span><span class="w"> </span><span class="s2">"join-pane -s '%%'"</span>
bind-key<span class="w"> </span>s<span class="w"> </span>command-prompt<span class="w"> </span>-p<span class="w"> </span><span class="s2">"send pane to:"</span><span class="w"> </span><span class="s2">"join-pane -t '%%'"</span>
</pre></div>
<p>This is the most popular binding for this 'moving' purposes. But - when you try
it entering window number as the destination (or source) you get a message:</p>
<blockquote>
<p>Can’t join a pane to its own window</p>
</blockquote>
<p>WTF?</p>
<p>Let's RTFM:</p>
<blockquote>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="nv">join</span><span class="o">-</span><span class="nv">pane</span><span class="w"> </span>[<span class="o">-</span><span class="nv">bdhv</span>]<span class="w"> </span>[<span class="o">-</span><span class="nv">l</span><span class="w"> </span><span class="nv">size</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">-</span><span class="nv">p</span><span class="w"> </span><span class="nv">percentage</span>]<span class="w"> </span>[<span class="o">-</span><span class="nv">s</span><span class="w"> </span><span class="nv">src</span><span class="o">-</span><span class="nv">pane</span>]<span class="w"> </span>[<span class="o">-</span><span class="nv">t</span><span class="w"> </span><span class="nv">dst</span><span class="o">-</span><span class="nv">pane</span>]
<span class="w"> </span><span class="ss">(</span><span class="nv">alias</span>:<span class="w"> </span><span class="nv">joinp</span><span class="ss">)</span>
<span class="w"> </span><span class="nv">Like</span><span class="w"> </span><span class="nv">split</span><span class="o">-</span><span class="nv">window</span>,<span class="w"> </span><span class="nv">but</span><span class="w"> </span><span class="nv">instead</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">splitting</span>
<span class="w"> </span><span class="nv">dst</span><span class="o">-</span><span class="nv">pane</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="nv">creating</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">new</span><span class="w"> </span><span class="nv">pane</span>,<span class="w"> </span><span class="nv">split</span><span class="w"> </span><span class="nv">it</span><span class="w"> </span><span class="nv">and</span>
<span class="w"> </span><span class="nv">move</span><span class="w"> </span><span class="nv">src</span><span class="o">-</span><span class="nv">pane</span><span class="w"> </span><span class="nv">into</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">space</span>.<span class="w"> </span><span class="nv">This</span><span class="w"> </span><span class="nv">can</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">used</span>
<span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">reverse</span><span class="w"> </span><span class="k">break</span><span class="o">-</span><span class="nv">pane</span>.
</pre></div>
</blockquote>
<p>Yes - so this gives us nothing.</p>
<p>So - lurking through the horizon of the internet I found
<a href="https://forums.pragprog.com/forums/242/topics/10533">this</a>:</p>
<p>So it looks like the full path takes 2 arguments: window_number.pane_number</p>
<p>So yes - bindings are still valid but instead entering single window number we
should enter exact location like <strong>1.3</strong> (assuming you got at least 2 panes in
1 window)</p>
<p>Yes - this was not that obvious</p>Piwik!2014-11-16T00:00:00+01:002014-11-16T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-11-16:2014/Nov/16/piwik/<p><center><img alt="Piwik" src="https://maciej.lasyk.info/images/piwik.png" /></center></p>
<h1>Piwik?</h1>
<p><a href="http://piwik.org">Piwik</a> is an open - source alternative to <a href="https://analytics.google.com">Google Analytics</a>.
You can read the <a href="http://smashinghub.com/google-analytics-vs-piwik-the-ultimate-comparison.htm">ultimate comparison
here</a>.</p>
<h1>Why change?</h1>
<p>I've been using Google Analytics since its beginning. Was I happy with it? Yes.
So why did I change it finally? I have couple of reasons:</p>
<ul>
<li>Because I didn't use 90% functionalities of GA</li>
<li>Because I have corporate - paranoia (watched too many movies, played too many
games). Don't want Google to control my data</li>
<li>Because I like hosting my services and scale those</li>
<li>Because I feel great when my way of hosting Piwik runs faster than Google
Analytics</li>
<li>Because I love learning; and every new service is about learning how to
maintain, scale and keep it running smoothly</li>
<li>Because it's fun to create yet another Ansible playbook</li>
<li>Because it's fun to host yet another service with Docker</li>
<li>Because it's fun to have yet another part of cluster (regarding Docker
cluster)</li>
<li>Because Piwik is 100% open - source</li>
</ul>
<h1>How?</h1>
<p>Oh it was simple. There are 2 easy (and fast) methods of rolling up Piwik
yourself:</p>
<ol>
<li><a href="https://github.com/ICTO/ansible-piwik">Piwik Ansible playbook</a> - it's quite
old and I had to fix couple of lines to make it working with 1.7.2 (created
already <a href="https://github.com/ICTO/ansible-piwik/pull/2">pull request</a> for those fixes)</li>
<li><a href="https://github.com/RyanKung/docker-piwik-mariadb">Dockerfile for Piwik /
MariaDB</a></li>
</ol>
<p>Just choose a way which fits you the most (or poke around your own one).</p>
<h1>Anything else?</h1>
<p>Just remember to make sure that Piwik content headers have gzip compression and
also caching enabled. Hosting own analytics site means many requests to the
public interfaces so those should be optimized very well.</p>Orchestrating Docker containers at scale - slides from Linux Autumn 20142014-11-10T00:00:00+01:002014-11-10T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-11-10:2014/Nov/10/orchestrating-docker-containers-at-scale-slides-from-linux-autumn-2014/<p><center><img alt="Jesien" src="https://maciej.lasyk.info/images/jesien-logo.png" /></center></p>
<p>That's it. <a href="http://www.jesien.org">Linux Autumn 2014</a> is over. This year I finally decided to give a talk
instead of only listening and networking.</p>
<p>The subject of this talk was <strong>Orchestrating Docker containers at scale</strong>. As
this was first time Docker ever appeared on Linux Autumn Conference I decided
to explain basics of those containers and later jump into deeper water or
orchestration.</p>
<p><center>
<iframe src="//www.slideshare.net/slideshow/embed_code/41323728" width="510"
height="420" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"
style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width:
100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> <strong> <a
href="//www.slideshare.net/d0cent/orchestrating-docker-containersatscale"
title="Orchestrating Docker containers at scale" target="_blank">Orchestrating
Docker containers at scale</a> </strong> from <strong><a
href="//www.slideshare.net/d0cent" target="_blank">Maciej Lasyk</a></strong>
</div>
</center></p>
<p>Thanks for all of you guys - it was really nice to hear so many good words
about the presentation! Hope to see you at <a href="www.meetup.com/Krakow-DevOps/">DevOPS meetups
@KRK</a>!</p>
<p>Btw - Szczyrk and it's area are really great for hiking, MTB and running.
Could't resist those mountains so had a <a href="http://www.strava.com/activities/216556430">30-km run on
Friday</a>:</p>
<p><center><img alt="Beskid Śląski" src="https://maciej.lasyk.info/images/beskid_slaski.jpg" /></center></p>Remark2014-10-29T00:00:00+01:002014-10-29T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-10-29:2014/Oct/29/remark/<p><center><img alt="VIM" src="https://maciej.lasyk.info/images/vim.jpg" /></center></p>
<p>One of the rules that stay assures my productivity is using same tools
for many tasks. That's why I've been looking for Powerpoint / LibreImpress replacement.</p>
<p>But it had to fullfill couple of requirements:</p>
<ul>
<li><a href="http://daringfireball.net/projects/markdown/">Markdown</a> based (so no need to learn new languages and could be created in VIM)</li>
<li>Devops way - no clicking and fancy UIs, rather working in terminal</li>
<li>No binary files so changes could be diffed easily (and VCSes make sense)</li>
<li>Easy embedding of images</li>
</ul>
<p>And it's here: <a href="http://remarkjs.com/">Remark</a>. Hosted on <a href="https://github.com/gnab/remark">GitHub</a>, simple to use, quite a community around.</p>
<p>Running those slideshows with use of:</p>
<div class="highlight"><pre><span></span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">SimpleHTTPServer</span> <span class="mi">8001</span>
</pre></div>Focus2014-10-27T00:00:00+01:002014-10-27T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-10-27:2014/Oct/27/focus/<p><center><img alt="Focus" src="https://maciej.lasyk.info/images/focus.jpg" /></center></p>
<h1>Focus</h1>
<p>In 2001 a friend of mine wrote me via IM one sentence which remains in my
memory up to now. It was about 1am and he wrote:</p>
<blockquote>
<p>I've done nothing constructive for the last 30 minutes. I'm going to bed.</p>
</blockquote>
<p>He lost his focus. He was tired and started to procrastinate. He was my mentor
in the terms of <a href="https://en.wikipedia.org/wiki/Getting_Things_Done">Getting Things
Done</a>.</p>
<p>I was a student that times. For me it was rather normal to work until falling
asleep on the keyboard. I just couldn't tell myself "enough, go to bed"..
That's why that sentence was so important for me.</p>
<h1>Pomodoro</h1>
<p>I find <a href="https://en.wikipedia.org/wiki/Pomodoro_Technique">Pomodoro</a> technique
very helpful when having problems with concentrating or my plate is totally
full. Just find your way through <strong>pomidori</strong> and after full set of 4 go
outside for 15-30 mins. That really helps.</p>
<h1>Limit your WIP</h1>
<p>That is taken from
<a href="http://www.agileweboperations.com/kanban-wip-limits-the-fine-art-of-focus">Kanban</a>.
I wouldn't be able to use Pomodoro having couple of tasks running
together. If I want to be efficient I have to <strong>remove any context switching</strong>
and work only on one task at a time. Kanban or Scrum ensures that I can focus
on one particular job.</p>
<h1>Remove distractors</h1>
<p>Using <a href="https://en.wikipedia.org/wiki/Xfce">XFCE</a> I can easily remove almost all
distractors simply by pressing <strong>F11</strong>. That gives me fullscreen mode of
current window hiding any blinking icons in the system bar.</p>
<p>Also I removed any focus - stealing apps etc (e.g. Skype tried to get my focus
when receiving new message etc).</p>
<p>Finally I learned to check all notifiers every single pomidori starts. So this
is the time I look at the calendar, e-mail, Skype and so on. Then I just put
down to the <a href="https://workflowy.com/">Workflowy</a> list of micro - tasks for this
day (reply e-mail, schedule meeting etc) and go on with the main job. So I can
use one of the pomidoris just for dealing with those micro - jobs (it's great
after lunch when feeling sleepy).</p>
<p>I also removed any notifications and LED blinking from all my cell - phones.
This was quite a huge time - saver. Even after work I just don't have this
feeling of checking that blinking LED. I just check my private e-mail like 5
times a day. It's enough.</p>
<h1>Take control</h1>
<p>Some time ago I felt like social networks were taking control over me. Having
Facebook app in the phone just made me poke around it almost unconsciously /
automatically. So instead of reading something valuable (like RSSes or Kindle)
I simply burnt my time on some bullshit. So I just removed all those apps from
my phone. It was fucking genius. No more bullshit, distracting myself etc. Time
saved - profit.</p>
<h1>Isolate</h1>
<p>Just buy yourself good head / earphones and use Spotify / Deezer. Working in an
open - space (or similiar) area is sometimes fatiguing.</p>
<h1>Win</h1>
<p>Yup - with the ability to focus on my job I simply win my time. And that gives
me oppurtunity to do a lot more. Great win for anyone that is interested in
almost all field of life ;)</p>
<p>But.. truly to be winners we have to practice all the time. Loosing focus is
very easy and kinda natural. So we have to control it. Good luck :)</p>Isolation2014-10-22T00:00:00+02:002014-10-22T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-10-22:2014/Oct/22/isolation/<h1>Isolation - what?</h1>
<p>Many of guys I talk to tend to run their Linux containers (LXC, Docker, you
name it) straight on bare. And I call this a bad idea. This is because when
running all the containers within one single box you will be hit by network or
security problems. Not today, but it will happen. Why?</p>
<h1>Clusterfuck network</h1>
<p>This term is used to describe a network where noone knows what's happening.
Frankly it's working but don't-try-to-change-that (unless you're stupid brave
/ enough).</p>
<p><center><img alt="Clusterfuck network" src="https://maciej.lasyk.info/images/clusterfucknetwork.jpg" /></center></p>
<p>How does it refers to containers? Imagine that above wires are iptables rules.
Should I write more? What happens when you restart iptables? Did you just
killed the network in 300 containers running on that box? Oh whatever iptables
- think docker0 interface + bonding/teaming + multiple bridges + veths + VIPs +
vlans + whatever-your-sysadmins-come-with</p>
<p>And now one single junior - sysadmin just reloads iptables as was written in
the 5th point of never-updated-procedure.</p>
<p>That's why I really like to have containers grouped on KVM VMs. KVM overhead is
minimal these years. And it can save your ass. Just make a proper Ha planning
and even if you kill some group of containers you will not have to firefight
having 5 managers simultaneously on 3 calls ;)</p>
<h1>Security</h1>
<p>Yes I know - this term is usually some big-fucking-unknown for most of guys /
ignorants. And that is why you should isolate groups of containers from each other. Throw
the first rock who uses SELinux / enforcing (under these circumstances you
could possibly cover the security subject in the terms on containers). But even
if you do then see > <strong>Clusterfuck network</strong> paragraph).</p>
<p>Btw - I had this talk during Infosec meetup some time ago about Docker
security - <a href="http://www.slideshare.net/d0cent/docker-rhel">enjoy</a>!</p>
<h1>What about IaaSes?</h1>
<p>Yes - if you're running <a href="https://openshift.github.io/">OpenShift</a> (or any other
Open: Nebula / Stack / Whatever) then probably you will not care about isolation at
all because this will be done for you without your interference. But -
seriously? Usually people rather use VMs for hosting nodes of IaaSes.</p>
<h1>Recap?</h1>
<p>No. Just wrote this after having a failure with LXC containers on Centos7 and
no-real-explanation just "have-you-tried-to-turn-it-off-and-on-again?".</p>
<p><center><img alt="Isolation" src="https://maciej.lasyk.info/images/isolation.jpg" /></center></p>Invitation for Linux Autumn 20142014-10-14T00:00:00+02:002014-10-14T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-10-14:2014/Oct/14/invitation-for-linux-autumn-2014/<p><center><img alt="Jesien" src="https://maciej.lasyk.info/images/jesien-logo.png" /></center></p>
<p>My speaker - break is over. Another major conference on the horizon. This time it's one of
the biggest Linux Conferences here in Poland.</p>
<p>I'll talk there about taking next step in automation of Linux containers
(<a href="http://docker.io">Docker</a> based) - cluster management based on <a href="https://github.com/coreos/fleet">Fleet</a>
and <a href="https://github.com/GoogleCloudPlatform/kubernetes">Kubernetes</a>. Also will
tell couple of words about <a href="http://www.fig.sh/">fig</a> and history of scaling webapps.</p>
<p>Still not sure if you should attend? Check this year's
<a href="http://www.jesien.org/2014/pl/agenda">agenda</a> and great
<a href="http://www.jesien.org/2014/en/hotel">location</a> of the event! Remember to take
a bike (great mtb and road trails there) or running shoes - this is one of the
best places to have a workout :)</p>
<p>See you there guys!</p>OpenWRT, Dnsmasq + external DNS2014-09-22T00:00:00+02:002014-09-22T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-09-22:2014/Sep/22/openwrt-dnsmasq-external-dns/<p>I use <a href="https://openwrt.org/">OpenWRT</a> on my home access - points instead of
stock software (due to security concerns and my OpenSourceTheWorld approach).</p>
<p>I also have <a href="https://en.wikipedia.org/wiki/BIND">BIND</a> in use on my
home-server, so I can use e.g. <a href="https://maciej.lasyk.info/2014/Sep/02/dns-views-in-bind/">zones</a>.</p>
<p>So I was trying to make OpenWRT send DNS IP address to DHCP clients. By default
OpenWRT will send it's own IP (it's using Dnsmasq onboard). And I just couldn't
find option to set this manually (using webinteface).</p>
<p>Finally I logged into CLI interface via SSH and found that this change can be
made in two ways.</p>
<p>In <strong>/etc/dnsmasq.conf</strong> by adding line:</p>
<div class="highlight"><pre><span></span>dhcp-option<span class="o">=</span><span class="m">6</span>,IP1,IP2
</pre></div>
<p>Or in <strong>/etc/config/dhcp</strong> by adding to the section <strong>config dnsmasq</strong>:</p>
<div class="highlight"><pre><span></span>list<span class="w"> </span><span class="s1">'dhcp_option'</span><span class="w"> </span><span class="s1">'6,192.168.1.29'</span>
</pre></div>
<p>Just restart Dnsmaq service afterwards and you're done.</p>Network Manager Dispatcher2014-09-14T11:46:00+02:002014-09-14T11:46:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-09-14:2014/Sep/14/network-manager-dispatcher/<p><a href="https://en.wikipedia.org/wiki/NetworkManager">NetworkManager</a> is often
associated with some voodoo - network - magic tool, that does the work on
laptop, but better switch it off on server. I won't argue about that, just
wanted to give you something I personally think make NM more usable.</p>
<p>So it is the <strong>/etc/NetworkManager/dispatcher.d</strong> - this is directory under
which lay scripts which will be executed on network change. Following
man-pages:</p>
<p><em>NetworkManager will execute scripts in the /etc/NetworkManager/dispatcher.d
directory in alphabetical order in response to network events. Each script
should be a regular executable file, owned by root.</em></p>
<p><em>Furthermore, it must not be writable by group or other, and not setuid.</em></p>
<p><em>Each script receives two arguments, the first being the interface name of
the device just activated, and second an action.</em></p>
<ul>
<li><strong>down</strong> - The interface has been deactivated.</li>
<li><strong>vpn-up</strong> - A VPN connection has been activated.</li>
<li><strong>vpn-down</strong> - A VPN connection has been deactivated.</li>
<li><strong>hostname</strong> - The system hostname has been updated. Use gethostname(2) to
retrieve it.</li>
<li><strong>dhcp4-change</strong> - The DHCPv4 lease has changed (renewed, rebound, etc).</li>
<li><strong>dhcp6-change</strong> - The DHCPv6 lease has changed (renewed, rebound, etc).</li>
</ul>
<p>So basically without manually detecting change in network events we may use the
tool that was created just for this particular goal.</p>Ansible Playbook for favtools2014-09-12T23:24:00+02:002014-09-12T23:24:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-09-12:2014/Sep/12/ansible-playbook-for-favtools/<h3>playbook</h3>
<p>This rather doesn't fit my github repo ;)</p>
<div class="highlight"><pre><span></span>-<span class="w"> </span>name:<span class="w"> </span>Ensure<span class="w"> </span>my<span class="w"> </span>favorite<span class="w"> </span>tools<span class="w"> </span>are<span class="w"> </span>installed
<span class="w"> </span>yum:<span class="w"> </span><span class="nv">pkg</span><span class="o">={{</span>item<span class="o">}}</span><span class="w"> </span><span class="nv">state</span><span class="o">=</span>latest
<span class="w"> </span>with_items:
<span class="w"> </span>-<span class="w"> </span>strace
<span class="w"> </span>-<span class="w"> </span>ltrace
<span class="w"> </span>-<span class="w"> </span>tcpdump
<span class="w"> </span>-<span class="w"> </span>vim
<span class="w"> </span>-<span class="w"> </span>procps
<span class="w"> </span>-<span class="w"> </span>sysstat
<span class="w"> </span>-<span class="w"> </span>git
<span class="w"> </span>-<span class="w"> </span>make
<span class="w"> </span>-<span class="w"> </span>gcc
<span class="w"> </span>-<span class="w"> </span>tmux
<span class="w"> </span>-<span class="w"> </span>links
<span class="w"> </span>-<span class="w"> </span>man
<span class="w"> </span>-<span class="w"> </span>ntp
<span class="w"> </span>-<span class="w"> </span>bind-utils
<span class="w"> </span>-<span class="w"> </span>nmap
<span class="w"> </span>-<span class="w"> </span>wget
<span class="w"> </span>-<span class="w"> </span>telnet
</pre></div>
<p>And there would be one <a href="http://docs.ansible.com/playbooks_roles.html#role-dependencies">dependency</a> here (<a href="https://fedoraproject.org/wiki/EPEL">EPEL</a> repo):</p>
<div class="highlight"><pre><span></span>-<span class="w"> </span>name:<span class="w"> </span>install<span class="w"> </span>epel<span class="w"> </span>repo
<span class="w"> </span>yum:<span class="w"> </span><span class="nv">name</span><span class="o">=</span>http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm<span class="w"> </span><span class="nv">state</span><span class="o">=</span>present
</pre></div>
<p>That is only for starters; I'd add here also <a href="http://www.sysdig.org/">sysdig</a> and whole playbook for preparing user env.</p>Sysadmin evolution2014-09-04T18:38:00+02:002014-09-04T18:38:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-09-04:2014/Sep/04/sysadmin-evolution/<h3>Sysadmin evolution</h3>
<p>A couple of days ago on <a href="http://training.linuxfoundation.org/sysadmin-evolution">The Linux Foundation</a>
appeared an image showing how could sysadmin advance his experience. I think
that it's really worth mentioning, so check this <a href="https://maciej.lasyk.info/images/EvolutionSysadmin_Infographic.png">image</a></p>DNS views in BIND2014-09-02T08:15:00+02:002014-09-02T08:15:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-09-02:2014/Sep/02/dns-views-in-bind/<h3>Why would I use DNS BIND views?</h3>
<p>From the manual: <strong>The view clause allows BIND to provide different
functionality based on the hosts accessing it.</strong></p>
<p>So basically you can provide one group of users with one variant of same zone
and different part of users with other variant of zone file.</p>
<p>This is very important - I've seen many poor - man's solutions when there was a
need to provide different views in DNSes. <strong>view</strong> clause is the proper
solution.</p>
<h3>How?</h3>
<p><strong>view Clause Syntax:</strong></p>
<div class="highlight"><pre><span></span>view<span class="w"> </span><span class="s2">"view_name"</span><span class="w"> </span><span class="o">[</span>class<span class="o">]</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span>match-clients<span class="w"> </span><span class="o">{</span><span class="w"> </span>address_match_list<span class="w"> </span><span class="o">}</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span>match-destinations<span class="w"> </span><span class="o">{</span><span class="w"> </span>address_match_list<span class="w"> </span><span class="o">}</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span>match-recursive-only<span class="w"> </span><span class="o">{</span><span class="w"> </span>yes<span class="w"> </span><span class="p">|</span><span class="w"> </span>no<span class="w"> </span><span class="o">}</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span>//<span class="w"> </span>view<span class="w"> </span>statements
<span class="w"> </span>//<span class="w"> </span>zone<span class="w"> </span>clauses
<span class="o">}</span><span class="p">;</span>
</pre></div>
<p>So i.e. when we want to serve different view for <strong>lasyk.info</strong> domain for 2
grups of users (one would be guys from internet, second - guys from my LAN) we
would do something like this:</p>
<p><strong>set up proper ACL:</strong></p>
<div class="highlight"><pre><span></span>acl<span class="w"> </span>internal<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="m">192</span>.168.10.0/24<span class="p">;</span>
<span class="w"> </span>localhost<span class="p">;</span>
<span class="o">}</span><span class="p">;</span>
</pre></div>
<p><strong>define zone:</strong></p>
<div class="highlight"><pre><span></span>view<span class="w"> </span><span class="s2">"internal-view"</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>match-clients<span class="w"> </span><span class="o">{</span><span class="w"> </span>internal<span class="p">;</span><span class="w"> </span><span class="o">}</span><span class="p">;</span>
<span class="w"> </span>recursion<span class="w"> </span>yes<span class="p">;</span>
<span class="w"> </span>zone<span class="w"> </span><span class="s2">"lasyk.info"</span><span class="w"> </span>IN<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">type</span><span class="w"> </span>master<span class="p">;</span>
<span class="w"> </span>file<span class="w"> </span><span class="s2">"internal.lasyk.info.conf"</span><span class="p">;</span>
<span class="w"> </span>allow-transfer<span class="w"> </span><span class="o">{</span><span class="w"> </span>any<span class="p">;</span><span class="w"> </span><span class="o">}</span>
<span class="w"> </span><span class="o">}</span><span class="p">;</span>
view<span class="w"> </span><span class="s2">"external-view"</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>match-clients<span class="w"> </span><span class="o">{</span><span class="w"> </span>any<span class="p">;</span><span class="w"> </span><span class="o">}</span><span class="p">;</span>
<span class="w"> </span>recursion<span class="w"> </span>no<span class="p">;</span>
<span class="w"> </span>zone<span class="w"> </span><span class="s2">"lasyk.info"</span><span class="w"> </span>IN<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">type</span><span class="w"> </span>master<span class="p">;</span>
<span class="w"> </span>file<span class="w"> </span><span class="s2">"external.lasyk.info.conf"</span><span class="p">;</span>
<span class="w"> </span>allow-transfer<span class="w"> </span><span class="o">{</span><span class="w"> </span>none<span class="p">;</span><span class="w"> </span><span class="o">}</span><span class="p">;</span>
<span class="w"> </span><span class="o">}</span><span class="p">;</span>
</pre></div>
<p>Now just create proper zone - files, <strong>rndc reload</strong> and test it with <strong>dig</strong></p>Discourse and Piwik2014-08-23T00:15:00+02:002014-08-23T00:15:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-08-23:2014/Aug/23/discourse-and-piwik/<h3>Erm, what?</h3>
<p>For the last couple of years I've been favorizing Open Source software. That's
why after many years of using <a href="https://www.disqus.com">Disqus</a> for comments and
<a href="https://analytics.google.com">Google Analytics</a> for statistics I've decided
to go Open Source.</p>
<h3>Why?</h3>
<ul>
<li>Because it's open source</li>
<li>Because I like hosting my services on my servers</li>
<li>Because I value my privacy</li>
<li>Because when I host something then it's on my rules</li>
</ul>
<h3>Discourse</h3>
<ul>
<li>This is Disqus alternative</li>
<li>Official webpage: <a href="http://www.discourse.org/">Discourse</a></li>
<li>Install notes via <strong>Docker</strong>(!): <a href="https://github.com/discourse/discourse/blob/master/docs/INSTALL.md">github</a></li>
</ul>
<h3>Piwik</h3>
<ul>
<li>This is Google analytics alternative</li>
<li>Official webpage: <a href="http://piwik.org/">Piwik</a></li>
<li>Install notes: <a href="http://piwik.org/docs/installation/">here</a></li>
</ul>
<p>Piwik doesn't have official Docker images - I'll work on that within <a href="https://github.com/fedora-cloud/Fedora-Dockerfiles">Fedora Dockerfiles</a>
and maybe I'll try to push this as another official installation method of Piwik.</p>Logout from Facebook2014-08-22T21:58:00+02:002014-08-22T21:58:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-08-22:2014/Aug/22/logout-from-facebook/<h3>Why?</h3>
<p>It could be a surprize that I won't mention here Facebook privacy policy. I'm
rather open person on the web, so I really don't care about anyone stalking my
actions (as I rather keep private matters outside the Internet). It's about
trade - offs when you start exploring the web and create profiles on social
services - you sell a bit of your privacy and get back the possibility to interact
with other people who you don't have to know or cannot contact personally.</p>
<p><strong>So - why?</strong></p>
<ul>
<li>I realized that 4k of unread RSSes are awating me (this number grew because
of my FB activity)</li>
<li>I get almost nothing productive from FB - just people status updates etc</li>
<li>I found that I've been poking around FB without any self - control loosing
too much time</li>
<li>I learn almost nothing from FB; percent of valuable posts is so low that
it's irrational to name it "profitable"</li>
<li>From the wider perspective - FB is just boring. I personally find interesting
only those informations that are educative so I can learn something new. Major
of FB updates is just rumour.</li>
<li>People tend to show themselves in the best possible situations, under bright
glare and we usually try to compare ourselves to them (this is how human is
constructed imo). And this can lead us to frustration. Why would we do this to
ourselves?</li>
</ul>
<h3>What will I miss?</h3>
<ul>
<li>Mainly contacts with people living far away</li>
<li>Some of companies instead of running own blog with RSS feed just use FB to
post updates on events, promotions etc</li>
</ul>
<h3>What will I get?</h3>
<ul>
<li>Time to read RSSes -> valuable articles, updates etc</li>
<li>More time to focus - less context - switching (yup, I caught myself on that
due to FB activity)</li>
<li>~100mb of space on my phone ;)</li>
<li>I'll focus on this blog so there's non - zero chance that it will be more
human - usable</li>
<li>I was always a writer kind of person who likes to document events. FB is very
bad when someones decide to post there story from a voyage because it will
simply be lost after some time. Internet of things works better - when you
have your own webpage it is indexed by many serch engines and people can easily
come back here.</li>
<li>Confidence that content I create will survive as it is hosted in a way I decide</li>
</ul>
<h3>Did I logout completely?</h3>
<p><strong>No</strong>. I uninstalled FB apps from my phone - thanks to that I just control my
level of dopamine during day :)</p>
<p>Moreover I removed login&pwd from the browser memory on my laptop - anytime I
want to check on FB I have to login manually. Lastly - I set up one-time-per-day
session of FB (~5min). Just to check on my family.</p>Online Courses2014-08-20T22:46:00+02:002014-08-20T22:46:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-08-20:2014/Aug/20/online-courses/<h3>What about onlince courses?</h3>
<p>I've been using couple of online educational services (IT mainly) and wanted
to share a short list of those I used - with a little comment:</p>
<ul>
<li><a href="https://www.coursera.org/">Coursera</a> - library full of varied courses, mainly
university level; ++ for mobile apps; courses are not self - paced</li>
<li><a href="https://www.codeschool.com/">CodeSchool</a> - Mainly webdev & mobiledev courses;
sometimes I have feeling that videos are too childish; also not that many
excercises to do; self - paced</li>
<li><a href="http://www.codecademy.com/">CodeAcademy</a> - like <strong>CodeSchool</strong> mainly webdev
courses (and backend ones also); not that childish - I liked using this;
self - paced</li>
<li><a href="http://www.learnstreet.com/">LearnStreet</a> - pity it's closed now; this was
really great service with many excercises and rich content; mentioning it
because I really believe it will be live again; self - paced</li>
<li><a href="https://www.udemy.com/">udemy</a> - didn't use this one, but my wife does ;)
self - paced, rich of wide angle - courses (not only IT)</li>
<li><a href="https://www.class-central.com">ClassCentral</a> - it's rather an agregator of
many courses - engines (like <strong>Coursera</strong>). Worth mentioning as provides
calendar of courses from many services</li>
<li><a href="http://www.pluralsight.com/">Pluralsight</a> - very rich library of self -
paced IT courses; I've moved here as this one covers majority of topics
that I wanted to learn about or retrain</li>
</ul>
<p>I think that those services could be a solid base for development planning
in any company.</p>Lectures during holidays2014-08-20T00:31:00+02:002014-08-20T00:31:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-08-20:2014/Aug/20/lectures-during-holidays/<h3>Reading It books during holidays?</h3>
<p><strong>Are you insanse?</strong> Someone could ask me this question, but seriously -
during holidays I read just whatever I have on my list; I don't divide lectures
into "workdays" and "holidays". I just read what I'd like to read and what I'm
comfortable with at the moment. And if those are IT books during holidays - it's
totally ok :)</p>
<h3>Mature optimization</h3>
<p><center><a href="http://carlos.bueno.org/optimization/"><img alt="Mature Optimization" src="https://maciej.lasyk.info/images/mature-optimization.jpg" /></a></center></p>
<p><a href="http://carlos.bueno.org/optimization/">Mature Optimization</a> by <strong>Carlos Bueno</strong>
is a book which could be described the best using words of the author:</p>
<p><em>I spent a good chunk of the last year working on an internal
training class and a short book about performance measurement
and optimization. My employer, Facebook, has kindly allowed me
to release the book for free.</em></p>
<h3>Docker Book</h3>
<p><center><a href="http://www.dockerbook.com/"><img alt="Docker Book" src="https://maciej.lasyk.info/images/dockerbook.jpg" /></a></center></p>
<p>This is a full know - how about Docker containers. Anyone that thinks about
turning his infrastructure in the way of lightweight containers should start
with this reading. Nice to see <a href="http://www.colliernotes.com/">Scott Collier</a>
as the technical reviewer. Scott works in <strong>RedHat</strong> on Openshift and Docker
(also <a href="https://github.com/GoogleCloudPlatform/kubernetes">Kubernetes</a>). I
collaborate with Scott on <a href="https://github.com/fedora-cloud/Fedora-Dockerfiles">Fedora Dockerfiles</a> project within <a href="https://fedoraproject.org">FedoraProject</a>.</p>
<p>This books imo lacks chapter covering cluster management subject, but as it
is (the book) still updated with new content I think that we'll have this
covered.</p>
<h3>Run or die</h3>
<p><center><a href="http://www.amazon.com/Run-Die-Kilian-Jornet/dp/1937715094"><img alt="Run or Die" src="https://maciej.lasyk.info/images/runordie.jpg" /></a></center></p>
<p>This is a story of one of the most famous skyrunners of our times. He still
amazes us with his new achievements. What can I say - this is a nice and easy
lecture I really wanted to read (thanks to my wife for finding this!)</p>
<h3>The Thriatlete's Training Bible</h3>
<p><center><a href="http://www.amazon.com/Triathletes-Training-Bible-Joe-Friel-ebook/dp/B00BUWY74M"><img alt="The Thriatlete's Training Bible #" src="https://maciej.lasyk.info/images/tritrainingbible.jpg" /></a></center></p>
<p>As I'm planning to start in my first thriatlon next year, all this year I'm
preparing for this event. The best way imo is to do the lesson and learn
about theory of proper training. I know the author <strong>John Friel</strong> for his
other books (especially <a href="http://www.amazon.com/Cyclists-Training-Bible-Joe-Friel/dp/1931382212">The Cyclist Training Bible</a>)
and I really found those lectures very rich about good practices, training
sessions examples and whole training theory.</p>
<h3>2 weeks</h3>
<p>...and couple ot thousands of pages to read & understand & make notes. Yeah ;)</p>Automating Pelican with Git2014-08-19T00:35:00+02:002014-08-19T00:35:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-08-19:2014/Aug/19/automating-pelican-with-git/<p><center><img alt="Pelican Automation with Git" src="https://maciej.lasyk.info/images/pelican-automation.png" /></center></p>
<p>This is a first blogpost of a whole series about working with
<a href="http://docs.getpelican.com/">Pelican</a>. I'll be publishing all future post with
the hashtag <a href="https://maciej.lasyk.info/tag/pelican.html">#pelican</a>.</p>
<h3>What kind of automation?</h3>
<p>My first step with migrating my <a href="http://maciek.lasyk.info/sysop">old blog</a> to
this <strong>Pelican - based</strong> was to automate the whole process of deploying new
posts. I just want to work with <strong>Markdown</strong> files on my laptop and push those
to the git repository. And let the rest magically happen in the background.</p>
<h3>Architecture</h3>
<p>Basically I write posts on my laptop, then I push changes to the remote bare
git repo, this repo has <strong>post-update</strong> hook which communicates with my
webserver and tells him to pull all the changes:</p>
<div class="highlight"><pre><span></span>laptop<span class="w"> </span>with<span class="w"> </span>markdowns<span class="w"> </span>----<span class="o">(</span>git<span class="w"> </span>push<span class="o">)</span>----><span class="w"> </span>remote<span class="w"> </span>git<span class="w"> </span>bare<span class="w"> </span>repo<span class="w"> </span>
----<span class="o">(</span>post-update<span class="w"> </span>hook----><span class="w"> </span>webserver
</pre></div>
<h3>Abracadabra</h3>
<p>What's important here - I've added to the repo my whole <strong>pelican</strong>
directory. This is because I'm realy lazy and just wanted to keep all the
changes and use this repo also for backup purposes. Of course you'll want to
play with <strong>.gitignore</strong> file and ommit couple of dirs and files :)</p>
<p>On the server with git bare repo I've created <strong>hooks/post-update</strong> file
(perms 0700) with following contents:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
<span class="c1"># just invoke the update script on the webserver:</span>
ssh<span class="w"> </span>home-docker<span class="w"> </span><span class="s1">'/home/docent/scripts/update_maciej.lasyk.info.sh'</span>
</pre></div>
<p>As you can see I'm using here <strong>ssh home-docker</strong> command. This is an alias
that exists in my <strong>~/.ssh/config</strong>:</p>
<div class="highlight"><pre><span></span>Host<span class="w"> </span>home-docker
Hostname<span class="w"> </span>vm-9-containers.netrunner.lasyk.info
User<span class="w"> </span>docent
Port<span class="w"> </span><span class="m">22</span>
ProxyCommand<span class="w"> </span>ssh<span class="w"> </span>-p<span class="w"> </span><span class="m">55555</span><span class="w"> </span>docent@netrunner.lasyk.info<span class="w"> </span>nc<span class="w"> </span>%h<span class="w"> </span>%p<span class="w"> </span><span class="m">2</span>><span class="w"> </span>/dev/null
IdentityFile<span class="w"> </span>~/.ssh/id_rsa
</pre></div>
<p>This basically connects to VM hosting Docker containers through the bastion
server (no direct SSH allowed to containers VM from outside or DMZ).</p>
<p>So now anytime I post something and commit it to the repo above script will be
invoked via SSH. And this calls another script - this time on webserver:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/bash</span>
<span class="nv">GIT</span><span class="o">=</span>/usr/bin/git
<span class="nb">cd</span><span class="w"> </span>/srv/docker_mounts/nginx/public_html/maciej.lasyk.info/
<span class="nv">$GIT</span><span class="w"> </span>fetch<span class="w"> </span>--all
<span class="nv">$GIT</span><span class="w"> </span>reset<span class="w"> </span>--hard<span class="w"> </span>docent/master
</pre></div>
<p>Above script fetches all data from the remote repository and then sets the
last commit as the HEAD. No webserver restart is needed here as those are only
static files :)</p>Network Warrior2014-08-18T17:54:00+02:002014-08-18T17:54:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-08-18:2014/Aug/18/network-warrior/<p><center><img alt="Network Warrior" src="https://maciej.lasyk.info/images/networkwarrior.jpg" /></center></p>
<p>Some time ago I bought an e-book <strong>Network Warrior</strong> by <strong>Gary A. Donahue</strong>. As this
is rather a long reading (more than 700 pages) I didn't have enough time to
finish it. But right now - on holidays - once more I went through table of
contents and last chapter "<strong>Avoiding Frustration</strong>" brought my attention. I just couldn't
resist from reading this right - away. Those couple of pages consists of
author's experiences from years when he was still learning how to work in
bigger companies (corporate sizes) and how to collaborate with others acting
as teamplayer.</p>
<p>Seriously - if you didn't read it yet - just read it. From table of contents:</p>
<ol>
<li>Avoiding frustration</li>
<li>Why Everything Is Messed Up</li>
<li>How To Sell Your Ideas To Management</li>
<li>When To Upgrade And Why<ol>
<li>The Dangers Of Upgrading</li>
<li>Valid Reasons To Upgrade</li>
</ol>
</li>
<li>Why Change Control Is Your Friend</li>
<li>How Not To Be A Computer Jerk<ol>
<li>Behavioral</li>
<li>Environmental</li>
<li>Leadership And Mentoring</li>
</ol>
</li>
</ol>
<p>Author described so many situations we face on daily basis in our work - and
what is important - he wrote how to react and behave so everybody is content
and our work is effective and goes in right direction. A totally "<strong>must read</strong>"
for everyone that have any doubts.</p>
<p>You can check the book details on <a href="http://www.amazon.com/Network-Warrior-Gary-A-Donahue/dp/1449387861">Amazon</a></p>Creator Bias2014-07-29T00:15:00+02:002014-07-29T00:15:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-07-29:2014/Jul/29/creator-bias/<p>Quote of the day: “Humans are bad at predicting the performance of complex
systems [...] Our ability to create large & complex systems fools us into
believing that we’re also entitled to understand them”</p>
<p>Taken from <a href="http://carlos.bueno.org/optimization/mature-optimization.pdf">"Mature Optimization Handbook"</a> by Carlos Bueno</p>IT night 2014 recap & slides2014-06-01T00:00:00+02:002014-06-01T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-06-01:2014/Jun/01/it-night-2014-recap-slides/<p><center><img alt="IT night" src="https://maciej.lasyk.info/images/it-night-2014.jpg" /></center></p>
<p>First of all – this is a coolest event badge ever. Seriously – it’s awesomeness is about zoundsss! Thanks guys again for that!</p>
<p>So this night is over, I think we all had already enough sleep and now we’re able to do some recap. First of all – hard to say for me but this was my first IT night; somehow it happened that I couldn’t make it earlier (7 years to narrow it as this was 8th edition – lol). What was this night about? Everything that was somehow connected to IT topics; there were gaming consoles, Linux Millionaire – like trivia, throwing HDDs challenge, assembling PC challenge etc – all the things we did and we love or hate :) And of course – there were many talks given by awesome people (lol – including me) ;) But honestly – every one person who attended this night was really awesome – We could talk a lot with almost everyone and did not get bored :)</p>
<p>So – if you weren’t there – regret this all 365 following days and register to the next edition – it’s really a must – be event :) And for organizers – guys – you did a tremendous and great job – thank you for that!</p>
<p>My talk was about living and working in terminal. You can find this on Slideshare (embedded below). I also created GitHub project where I will develop those topics through the time. Wanna help? No prob – just ping me somehow ;)</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/1I8PZ52Sy9RymK" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe>
<p><div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/d0cent/ghost-intheshell" title="Ghost in the shell" target="_blank">Ghost in the shell</a> </strong> from <strong><a href="https://www.slideshare.net/d0cent" target="_blank">Maciej Lasyk</a></strong> </div></p>Atmosphere Conference 2014 recap2014-05-26T00:00:00+02:002014-05-26T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-05-26:2014/May/26/atmosphere-conference-2014-recap/<p><center><img alt="Atmo 2014" src="https://maciej.lasyk.info/images/atmo-2014.jpg" /></center></p>
<p>Atmosphere conference was a blast! Those 2 days were full of talking, arguing and thinking. Venue was really great – whole event took place in the Praha Cinema in Warsaw and we had that feeling of #RedCarpet party ;)</p>
<p>The Atmosphere Conference just like Velocity is all about scalability and performance so DevOPS techniques and culture are all around the place – you can literally feel those in the air. Also the event is about people you meet there – and as the afterparty kept the Conference standards (great organization, full of interesting people) I can only say, that this is “must be” for all our polish DevOPS and others that are taking care of scaling and polishing their platforms performance.</p>
<p>Someone asked me to compare Velocity and Atmosphere. Well – the latter one is generally polish Conference where 95% of participants are from Poland; yet this year lectures were mainly in english. And about technological part – this year there were guys from Akamai, Facebook, E-Bay, Chef, Percona, Prezi, Microsoft, some polish local businesses and me – from Fedora Project. Looks like this conference is going global, yup? :)</p>
<p>Oh and that running part – I stayed in a hotel just near to Lazienki Park – one of the most beautiful city – parks in the Poland. So 6 am I was already up & running through that alleys – I really recommend this place to every of you that wants to keep in motion ;)</p>
<p>I’m really happy to had been part of the Atmosphere 2014. My lecture was about scaling and securing node.js apps. Check those out:</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/Auh2Lp4o2QxZRG" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe>
<p><div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/d0cent/scaling-and-securing-nodejs-apps-35127655" title="Scaling and securing node.js apps" target="_blank">Scaling and securing node.js apps</a> </strong> from <strong><a href="//www.slideshare.net/d0cent" target="_blank">Maciej Lasyk</a></strong> </div></p>Invitation to AtmosphereConf2014-05-12T00:00:00+02:002014-05-12T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-05-12:2014/May/12/invitation-to-atmosphereconf/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/atmosphere-2014-300x168.jpg" /></center></p>
<p>I’ll give a talk at <a href="https://atmosphere-conference.com/">AtmosphereConf</a> – this event is all about performance and scalability, the biggest one in Poland of this kind. It remembers me about VelocityConf that I attended couple of years ago.</p>
<p>I’ll be there as a part of Fedora Project. And here is the abstract of my talk:</p>
<p>Scaling and securing node.js apps
After few years of node.js in operation we know it’s very fast. We all heard stories about backend services running with the help of node.js and V8 core. But what we have learned about the security of such applications? What are the major threats here? I’ll explain how to create node.js apps in a secure and reliable way. Also – I’ll show how those could be scaled easily with (or without) help of Linux containers (Docker based) or jail – systems like Selinux Sandbox or libvirt sandbox.</p>
<p>There’s still possibility to <a href="https://rejestracja.proidea.org.pl/registration/form.html?conferenceId=BD2C369D83E5939D36537511DF9E5D81">register</a> – and this event is really full of great people and talks – just can’t wait till I’m there :)</p>node.js security slides from SEConference 20142014-05-10T00:00:00+02:002014-05-10T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2014-05-10:2014/May/10/nodejs-security-slides-from-seconference-2014/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/node-js-S.jpg" /></center></p>
<p>Yesterday I gave talk about node.js security during SEConference here in Kraków. It was fun – especially afterparty where we could do some nerd – talking about our security concerns :)</p>
<p>Oh and that part:</p>
<p>me: “- hey, where did you get this Dragon Sector t-shirt?”</p>
<p>vnd: “- because I’m the member of Dragon sector”</p>
<p>me: “- LOL, that explains everything”</p>
<p>And I was wondering why this guy is called “vnd” – just like that one from Dragon Sector. It was so obvious – lolme ;)</p>
<p>Ok so now – slides:</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/gSGNb8sEtqoBZu" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe>
<p><div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/d0cent/nodejs-security" title="Node.js security" target="_blank">Node.js security</a> </strong> from <strong><a href="//www.slideshare.net/d0cent" target="_blank">Maciej Lasyk</a></strong> </div></p>11 Linux Session recap2014-04-09T20:40:00+02:002014-04-09T20:40:00+02:00docenttag:maciej.lasyk.info,2014-04-09:2014/Apr/09/11-linux-session-recap/<p><center><img alt="11 Linux Session" src="https://maciej.lasyk.info/images/11-300x89.jpg" /></center></p>
<p>11th edition of Linux Session is over. It was 2 days full of great
presentations, discussions and liters of beer we drank. I really enjoyed
"Self - hosting in practice" by <em>Błażej „Mewp” Święcicki. </em>I just felt
like this guys was talking about me, my home - server and my solutions
:) Unfortunately I didn't make it to "Linux namespaces: not only LXC"
by <em>Wirkijowski Wojciech</em> as I was late due to some accident on highway
which caused my delay.</p>
<p>Also I gave 2 talks during this conference - honestly it was my first
time to give 2 speeches during one event. It wasn't that easy to
prepare, but in the end I felt very good that I had that chance :) See
yourself the slides:</p>
<iframe width="427" height="356" style="border: 1px solid #CCC; border-width: 1px 1px 0; margin-bottom: 5px; max-width: 100%;" src="http://www.slideshare.net/slideshow/embed_code/33338110" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" allowfullscreen></iframe>
<iframe width="427" height="356" style="border: 1px solid #CCC; border-width: 1px 1px 0; margin-bottom: 5px; max-width: 100%;" src="http://www.slideshare.net/slideshow/embed_code/33338702" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" allowfullscreen></iframe>Docker vs LXC/Ansible?2014-03-16T00:00:00+01:002014-03-16T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-03-16:2014/Mar/16/docker-vs-lxcansible/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/containers.png" /></center></p>
<h3>Why this question?</h3>
<p>During last DevOPS meetup <a href="https://twitter.com/GrzegorzNosek">@GrzegorzNosek</a> asked very good question – why should one use Docker instead of pure LXC/Ansible?</p>
<p>Honestly I’ve been trying to answer myself this question for a while. I did in some part (included this in my talk I gave during that meetup: <a href="http://www.slideshare.net/d0cent/docker-rhel">http://www.slideshare.net/d0cent/docker-rhel</a>); while it’s about developers running development envs Docker is just so much easier to use.</p>
<p>But how should I explain using Docker for myself? I’m sysadmin and I love low-level – so LXC for me is just natural way of doing things :)</p>
<h3>What’s the difference for me?</h3>
<p>One thing you should know about me – I’m contributing to FedoraProject; lately I’ve been poking around Fedora-Dockerfiles project (https://git.fedorahosted.org/cgit/dockerfiles.git/) – I’m doing it for fun and also I wanted to learn more about Docker as I’m running some Open-Source projects with friends and had to find a easy way for them to rollup own development envs. Docker is the answer in this case.</p>
<p>So – currently I’m using Docker to prepare dev-envs for guys who knows nothing about DevOPS / Sysoping; writing Dockerfiles is so much fun (and sometimes so big hell :) ). And LXC? Together with Ansible I’m managing some servers’ resources (like VPN, DNS, some webservices etc). It’s also fun, it’s fast, rather reliable and it makes things so much easy to live with.</p>
<h3>So any winners here?</h3>
<p>But still – for me as guy who use rather fdisk than gparted (or virsh than virt-manager ;) ) Docker is not the case for managing services. And honestly I’m still looking for an answer for the question from subject of this blogpost. For now after couple of weeks poking around Docker (and months with LXC) I can tell this one obvious thing that when You know LXC than Docker is just so easy (e.g. running some daemons inside spartan-like Docker images can be a tough fight whe some libs or dependencies are missing). Also creating and running Dockerfiles is very easy – just like creating Ansible playbooks.</p>
<p>I think that I’m gonna do this one thing that I did couple of years ago when XEN and KVM were running shoulder to shoulder in the FOSS full-virt race. I’m just gonna use them both – Docker and LXC and see how things will develop. Docker is very great and easy to manage apps only (so Continuous Development with Docker is killing feature) and I’ll LXC/Ansible within some basic services (GitLab, DNS, VPN etc). But for more fun – I’m gonna keep both tracks, so e.g. when deploying GitLab within LXC I’ll create also Dockerfile for this.</p>
<p>This way I think that I will have a really good answer in just a couple of weeks and this should be nice subject for some conference talk?</p>
<p>Follow my <a href="https://github.com/docent-net">GitHub account</a> (or even better – <a href="https://twitter.com/docent_net">Twitter</a>) – I’ll post there updates about new playbooks and Dockerfiles.</p>Stop disabling SELinux – slides from my talk @ Krakow Infosec meetup2014-03-13T00:00:00+01:002014-03-13T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-03-13:2014/Mar/13/stop-disabling-selinux-slides-from-my-talk-krakow-infosec-meetup/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/post-38-pinguim-do-selinux.jpg" /></center></p>
<p>Every time you run setenforce 0, you make Dan Walsh weep. Dan is a nice guy and he certainly doesn’t deserve that. So see how not to disable SELinux and why it is so great :)</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/9mMyW30WRpJqX9" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe>
<p><div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/d0cent/selinux-single" title="Stop disabling SELinux!" target="_blank">Stop disabling SELinux!</a> </strong> from <strong><a href="//www.slideshare.net/d0cent" target="_blank">Maciej Lasyk</a></strong> </div></p>Infosec meetup invitation2014-03-06T00:00:00+01:002014-03-06T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-03-06:2014/Mar/06/infosec-meetup-invitation/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/cyber-attacks-hicube-infosec-300x168.jpg" /></center></p>
<p>Next week there will be first Infosec group meetup. It’ll take place in BaseLab here in Kraków. I’ll give a #SELinux presentation there and that will be the beginning of the whole SELinux series. I’ll try to enlight all of you who still “setenforce 0”.</p>
<p>You can RSVP here: <a href="http://www.meetup.com/Infosec-Krakow/events/168455142/">http://www.meetup.com/Infosec-Krakow/events/168455142/</a></p>
<p>And remember one, very important thing, Dan is watching you!</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/o5snlP8Y5GY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>RHEL/Fedora + Docker (and SELinux)2014-02-26T00:00:00+01:002014-02-26T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-02-26:2014/Feb/26/rhelfedora-docker-and-selinux/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/docker.png" /></center></p>
<p>It’s still fresh – I gave this talk a few hours ago at DevOPS meetup here in Krakow. There’s Dan Walsh’s surprise included ;)</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/4sT9375823WPty" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe>
<p><div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/d0cent/docker-rhel" title="RHEL/Fedora + Docker (and SELinux)" target="_blank">RHEL/Fedora + Docker (and SELinux)</a> </strong> from <strong><a href="//www.slideshare.net/d0cent" target="_blank">Maciej Lasyk</a></strong> </div></p>High Availability Explained2014-01-29T00:00:00+01:002014-01-29T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2014-01-29:2014/Jan/29/high-availability-explained/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/DevOpsDays-300x212.png" /></center></p>
<p>Yesterday’s DevOPS meetup was great – we talked a <em>lot</em>, saw a few terrific talks, drank beer (water in my case :( ) & ate plenty of pizza ;) I gave talk about HA – you can see slides below:</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/llGzVrHy4HPFQ9" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe>
<p><div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/d0cent/high-availability-explained" title="High Availability (HA) Explained" target="_blank">High Availability (HA) Explained</a> </strong> from <strong><a href="//www.slideshare.net/d0cent" target="_blank">Maciej Lasyk</a></strong> </div></p>Changing the World One Line of Code at a Time2013-12-29T00:00:00+01:002013-12-29T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2013-12-29:2013/Dec/29/changing-the-world-one-line-of-code-at-a-time/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/opensource-550x475-t-300x259.png" /></center></p>
<p>So – 2013 year is almost over. It’s time for recap. It’s time to ask a question – what have I done for The Community?</p>
<p>I’ve been a programmer for the past 15+ years and sysop for 12+. I always used free and open source software – like those from Apache project, perl, php, python etc. And I’ve been using Linux for the past 16+ years. And this is this word about which I’ve been thinking quite a lot lately: “using”. Three years ago I took part in the LinuxCon Europe conference (2011) at Prague. I met there many people, that inspired me by the way they acted.</p>
<p>Until then when I hit some issue with software I barely reported it via some Bugzilla to some project keepers. Barely… But then I realized that those guys’ way is great. Something is not working like You’d like it to? git clone the repo, change, create pull request… wait – maybe first meet guys and join project?</p>
<p>And that’s what I did this year. I joined two great projects: <a href="https://fedoraproject.org/">Fedora project</a> & <a href="https://www.gnome.org/">Gnome</a> project And started contributing. How much time does it cost me? It depends – there’re weeks when I do nothing, and there are those where I put 10 hours of effort to solve some problem or create new functionality. And it feels really good :)</p>
<p>So are you a contributor? No? Think about it! You don’t have to be a developer. Check this page of Gnome project: https://www.gnome.org/get-involved/ – you can help – and you can make a difference :)</p>KVM L2 filtering / virsh nwfilter2013-12-01T00:00:00+01:002013-12-01T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2013-12-01:2013/Dec/01/kvm-l2-filtering-virsh-nwfilter/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/kvmbanner-logo2.png" /></center></p>
<p>A few days ago while deploying another KVM host (this time in Hetzner.de datacenter) I had to lurk into deep networking internals. Hetzner has port security enabled on switches’ ports so there’s no way to use classical L2 bridging in netfilter. But I’ll write another post about resolving this one (yup, I did it – might be also usable for OVH users) ;)</p>
<p>This time I wanted to write a short post about network security in KVM host. Especially about ARP/IP spoofing. Problem? By default VMs can easily attack each other by spoofing each others MAC / IP addrs. Normally those type of attacks are mitigated on L2 – so we use e.g. port security, storm control, secure-arp-table and so on (sorry Juniper, I’m pure Cisco). So we know that L2 switch can be easily simulated on software side with netfilter / bridging. It’s easy to create network bridge, but it’s harder to create security policy for L2. And aAll that has to be done is to turn on ebtables and create some rules.</p>
<p>And here KVM / libvirt appears as very helpful. Writing ebtables rules is not a rocket science, but when managing multiple VMs it’s really easy to handle those with some higher – level tool. I ended up adding some rules to VMs’ XML definitions:</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="nt"><interface</span><span class="w"> </span><span class="na">type=</span><span class="s">'network'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><mac</span><span class="w"> </span><span class="na">address=</span><span class="s">'52:54:00:xx:yy:zz'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><source</span><span class="w"> </span><span class="na">network=</span><span class="s">'routed'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><model</span><span class="w"> </span><span class="na">type=</span><span class="s">'virtio'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><filterref</span><span class="w"> </span><span class="na">filter=</span><span class="s">'clean-traffic'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><parameter</span><span class="w"> </span><span class="na">name=</span><span class="s">'IP'</span><span class="w"> </span><span class="na">value=</span><span class="s">'88.99.11.22'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"></filterref></span>
<span class="w"> </span><span class="nt"><address</span><span class="w"> </span><span class="na">type=</span><span class="s">'pci'</span><span class="w"> </span><span class="na">domain=</span><span class="s">'0x0000'</span><span class="w"> </span><span class="na">bus=</span><span class="s">'0x00'</span><span class="w"> </span><span class="na">slot=</span><span class="s">'0x05'</span><span class="w"> </span><span class="na">function=</span><span class="s">'0x0'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"></interface></span>
</pre></div>
<p>So above you can see the “clean traffic” filter. What is that? Here a little explanation:</p>
<div class="highlight"><pre><span></span>root@cubryna<span class="w"> </span>qemu]#<span class="w"> </span>virsh<span class="w"> </span>nwfilter-dumpxml<span class="w"> </span>clean-traffic
<span class="nt"><filter</span><span class="w"> </span><span class="na">name=</span><span class="s">'clean-traffic'</span><span class="w"> </span><span class="na">chain=</span><span class="s">'root'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><uuid></span>2c7bbeb3-1438-zzz-yyy-xxx<span class="nt"></uuid></span>
<span class="w"> </span><span class="nt"><filterref</span><span class="w"> </span><span class="na">filter=</span><span class="s">'no-mac-spoofing'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><filterref</span><span class="w"> </span><span class="na">filter=</span><span class="s">'no-ip-spoofing'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><rule</span><span class="w"> </span><span class="na">action=</span><span class="s">'accept'</span><span class="w"> </span><span class="na">direction=</span><span class="s">'out'</span><span class="w"> </span><span class="na">priority=</span><span class="s">'-650'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><mac</span><span class="w"> </span><span class="na">protocolid=</span><span class="s">'ipv4'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"></rule></span>
<span class="w"> </span><span class="nt"><filterref</span><span class="w"> </span><span class="na">filter=</span><span class="s">'allow-incoming-ipv4'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><filterref</span><span class="w"> </span><span class="na">filter=</span><span class="s">'no-arp-spoofing'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><rule</span><span class="w"> </span><span class="na">action=</span><span class="s">'accept'</span><span class="w"> </span><span class="na">direction=</span><span class="s">'inout'</span><span class="w"> </span><span class="na">priority=</span><span class="s">'-500'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><mac</span><span class="w"> </span><span class="na">protocolid=</span><span class="s">'arp'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"></rule></span>
<span class="w"> </span><span class="nt"><filterref</span><span class="w"> </span><span class="na">filter=</span><span class="s">'no-other-l2-traffic'</span><span class="nt">/></span>
<span class="w"> </span><span class="nt"><filterref</span><span class="w"> </span><span class="na">filter=</span><span class="s">'qemu-announce-self'</span><span class="nt">/></span>
<span class="nt"></filter></span>
</pre></div>
<p>So basically “clean traffic” is a group of predefined filter references. Please read the libvirt documentation for details. Brief explanation would be: if clean traffic is applied on VM than such an VM will not be able to spoof MAC or IP addr (and some more rules as you can see above).</p>
<p>One could ask – why the heck didn’t I configured DHCP and instead of that I put static IP addr into VM XML config file? So – DHCP is great, but when you want to enable migration for VMs than before new host learns new VMs IP addr / MAC this VM can easily spoof it. So – it’s better to place IP into XML file.</p>
<p>Reference:</p>
<ul>
<li>http://libvirt.org/formatnwfilter.html</li>
<li>https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html-single/Virtualization_Deployment_and_Administration_Guide/index.html</li>
<li>https://events.linuxfoundation.org/slides/2011/lfcs/lfcs2011_cloud_kashyap.pdf</li>
<li>https://www.berrange.com/tags/ip-spoofing/</li>
</ul>Adding Python support for Ganglia - gmond on Centos / RHEL2013-11-10T14:40:00+01:002013-11-10T14:40:00+01:00docenttag:maciej.lasyk.info,2013-11-10:2013/Nov/10/adding-python-support-for-ganglia-gmond-on-centos-rhel/<p><center><img alt="Ganglia" src="https://maciej.lasyk.info/images/ganglia_logo_small.jpg" /></center></p>
<p>While creating Ganglia RPM packages for my Centos repository I got stuck
with a lack of Python support in Ganglia - gmond. I'll try to to explain
details of this whole issue.</p>
<p>First of all one could just install ganglia-gmond packages from
<a href="https://fedoraproject.org/wiki/EPEL">Epel</a> repo - and that will make
the general case. I checked EPEL packages and those contain
<strong>modpython.so</strong> so go for it ;) But I just couldn't go that way as I
had to create my own packages.</p>
<p>First of all - let's check the compilation params regarding to Python:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@vm-2-repo<span class="w"> </span>ganglia-gmond-3.6.0<span class="o">]</span><span class="c1"># ./configure --help | grep -i python</span>
<span class="w"> </span>--disable-python<span class="w"> </span>exclude<span class="w"> </span>mod_python<span class="w"> </span>and<span class="w"> </span>support<span class="w"> </span><span class="k">for</span><span class="w"> </span>metric<span class="w"> </span>modules<span class="w"> </span>written<span class="w"> </span><span class="k">in</span><span class="w"> </span>python
<span class="w"> </span>--with-python<span class="o">=</span>PATH<span class="w"> </span>Specify<span class="w"> </span>prefix<span class="w"> </span><span class="k">for</span><span class="w"> </span>python<span class="w"> </span>or<span class="w"> </span>full<span class="w"> </span>path<span class="w"> </span>to<span class="w"> </span>interpreter
</pre></div>
<p>So by default there should be Python support builtin. But after
compiling & building there's no <strong>modpython.so</strong> library
on <strong>/usr/lib64/ganglia</strong> and that is weird. Digging deeper
in <strong>config.log</strong> there's:</p>
<div class="highlight"><pre><span></span>configure:11482:<span class="w"> </span>checking<span class="w"> </span><span class="k">for</span><span class="w"> </span>python
configure:11500:<span class="w"> </span>found<span class="w"> </span>/usr/bin/python
configure:11512:<span class="w"> </span>result:<span class="w"> </span>/usr/bin/python
configure:11528:<span class="w"> </span>checking<span class="w"> </span>Python<span class="w"> </span>version
configure:11532:<span class="w"> </span>result:<span class="w"> </span><span class="m">2</span>.6
configure:11554:<span class="w"> </span>checking<span class="w"> </span>Python<span class="w"> </span>support
configure:11556:<span class="w"> </span>result:<span class="w"> </span>no
</pre></div>
<p>Hmm - that's really weird. Let's go even more deeper - now
with <strong>configure.ac</strong>:</p>
<div class="highlight"><pre><span></span>AC_ARG_WITH<span class="o">(</span><span class="w"> </span>python,
<span class="o">[</span><span class="w"> </span>--with-python<span class="o">=</span>PATH<span class="w"> </span>Specify<span class="w"> </span>prefix<span class="w"> </span><span class="k">for</span><span class="w"> </span>python<span class="w"> </span>or<span class="w"> </span>full<span class="w"> </span>path<span class="w"> </span>to<span class="w"> </span>interpreter<span class="o">]</span>,
<span class="o">[</span><span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>x<span class="s2">"</span><span class="nv">$withval</span><span class="s2">"</span><span class="w"> </span>!<span class="o">=</span><span class="w"> </span>xno<span class="p">;</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="nv">enable_python</span><span class="o">=</span><span class="s2">"yes"</span><span class="p">;</span><span class="w"> </span><span class="nv">PYTHON_BIN</span><span class="o">=</span><span class="s2">"</span><span class="nv">$withval</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="k">fi</span><span class="o">])</span>
<span class="o">[</span>...<span class="o">]</span>
AC_ARG_ENABLE<span class="o">(</span><span class="w"> </span>python,
<span class="o">[</span><span class="w"> </span>--disable-python<span class="w"> </span>exclude<span class="w"> </span>mod_python<span class="w"> </span>and<span class="w"> </span>support<span class="w"> </span><span class="k">for</span><span class="w"> </span>metric<span class="w"> </span>modules<span class="w"> </span>written
<span class="w"> </span><span class="k">in</span><span class="w"> </span>python<span class="o">]</span>,
<span class="o">[</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>x<span class="s2">"</span><span class="nv">$enableval</span><span class="s2">"</span><span class="w"> </span>!<span class="o">=</span><span class="w"> </span>xyes<span class="p">;</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="nv">enable_python</span><span class="o">=</span><span class="s2">"no"</span><span class="p">;</span><span class="w"> </span><span class="k">fi</span><span class="w"> </span><span class="o">]</span>,<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="nv">enable_python</span><span class="o">=</span><span class="s2">"yes"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">)</span>
</pre></div>
<p>Ok so it's true - by default there should be Python support. Let's check
further (still <strong>configure.ac</strong>):</p>
<div class="highlight"><pre><span></span><span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>x<span class="s2">"</span><span class="nv">$enable_python</span><span class="s2">"</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>xyes<span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>Checking<span class="w"> </span><span class="k">for</span><span class="w"> </span>python
<span class="w"> </span><span class="c1"># check for Python executable</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>-z<span class="w"> </span><span class="s2">"</span><span class="nv">$PYTHON_BIN</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>AC_PATH_PROG<span class="o">(</span>PYTHON_BIN,<span class="w"> </span>python<span class="o">)</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$PYTHON_BIN</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nv">PYTHON_BIN</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PYTHON_BIN</span><span class="s2">/bin/python"</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>-n<span class="w"> </span><span class="s2">"</span><span class="nv">$PYTHON_BIN</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="c1"># find out python version</span>
<span class="w"> </span>AC_MSG_CHECKING<span class="o">(</span>Python<span class="w"> </span>version<span class="o">)</span>
<span class="w"> </span><span class="nv">PyVERSION</span><span class="o">=</span><span class="sb">`</span><span class="nv">$PYTHON_BIN</span><span class="w"> </span>-c<span class="w"> </span><span class="o">[</span><span class="s1">'import sys; print sys.version[:3]'</span><span class="sb">`</span><span class="o">]</span>
<span class="w"> </span><span class="nv">PyMAJVERSION</span><span class="o">=</span><span class="sb">`</span><span class="nv">$PYTHON_BIN</span><span class="w"> </span>-c<span class="w"> </span><span class="o">[</span><span class="s1">'import sys; print sys.version[:1]'</span><span class="sb">`</span><span class="o">]</span>
<span class="w"> </span>AC_MSG_RESULT<span class="o">(</span><span class="nv">$PyVERSION</span><span class="o">)</span>
<span class="w"> </span><span class="nv">PYTHON_VERSION</span><span class="o">=</span><span class="nv">$PyVERSION</span>
<span class="w"> </span>AC_SUBST<span class="o">(</span>PYTHON_VERSION<span class="o">)</span>
<span class="w"> </span><span class="nv">PyEXEC_INSTALLDIR</span><span class="o">=</span><span class="sb">`</span><span class="nv">$PYTHON_BIN</span><span class="w"> </span>-c<span class="w"> </span><span class="s2">"import sys; print sys.exec_prefix"</span><span class="sb">`</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>-f<span class="w"> </span><span class="s2">"</span><span class="nv">$PyEXEC_INSTALLDIR</span><span class="s2">/include/python/Python.h"</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nv">PYTHON_INCLUDES</span><span class="o">=</span><span class="s2">"-I</span><span class="nv">$PyEXEC_INSTALLDIR</span><span class="s2">/include/python"</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>-f<span class="w"> </span><span class="s2">"</span><span class="nv">$PyEXEC_INSTALLDIR</span><span class="s2">/include/python</span><span class="nv">$PyVERSION</span><span class="s2">/Python.h"</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nv">PYTHON_INCLUDES</span><span class="o">=</span><span class="s2">"-I</span><span class="nv">$PyEXEC_INSTALLDIR</span><span class="s2">/include/python</span><span class="nv">$PyVERSION</span><span class="s2">"</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nv">PYTHON_INCLUDES</span><span class="o">=</span><span class="s2">""</span>
<span class="w"> </span><span class="nv">enable_python</span><span class="o">=</span><span class="s2">"no"</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span>AC_SUBST<span class="o">(</span>PYTHON_INCLUDES<span class="o">)</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nv">enable_python</span><span class="o">=</span><span class="s2">"no"</span>
<span class="w"> </span><span class="k">fi</span>
<span class="k">fi</span>
</pre></div>
<p>Ah crap - and here it is. Python support will be disabled and
modpython.so library won't be created when there's no header files for
Python on our OS. You can make sure if there is <strong>Python.h by using
locate / find / whatever.</strong></p>
<p>Solution? Just:</p>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>install<span class="w"> </span>python-devel
</pre></div>
<p>And that will make
it.</p>Samsung R&D Linux Boot competition – smashed ;) / Concepts github repo created!2013-10-29T00:00:00+01:002013-10-29T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2013-10-29:2013/Oct/29/samsung-rd-linux-boot-competition-smashed-concepts-github-repo-created/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/Samsung-booth-11-520x245-300x141.jpg" /></center></p>
<p>During last Linux Autumn conference (check my recap here) I won competition run by guys from the platinum sponsor – Samsung R&D Centre. There were two topics to be covered – each on other interesting aspect of Linux tuning. First one was about polishing Linux security and the second – about speeding up the Linux boot process. I wrote answers to both topics, but the latter was the winning one.</p>
<p>I was asked about my ideas that I used during that competition a few times until then. So after yet another conversation I thought, that maybe it’s time to write those and make them public – why not – I didn’t discover the America – just wrote down some known facts ;)</p>
<p>So – I strongly believe that github is a good place to publish things like that. I created ‘concepts’ repo where I’ll push any ideas that in my opinion is worth of it. Never mind if those are my own concepts or I heard those on some conference. This way I’m publishing my “concepts” notebook from my Evernote.</p>
<p>Feel free to clone this repo here: https://github.com/docent-net/concepts</p>
<p>The competition winning idea is here: https://github.com/docent-net/concepts/tree/master/booter</p>
<p>And one more thing. This competition was awarded with this beauty Samsung Galaxy SIII (yes, it’s only box; I already have exactly the same phone, so this one is landing on E-Bay):</p>
<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/1382195_10151717010150911_753129479_n-300x300.jpg" /></center></p>Linux Autumn 2013 – recap2013-10-13T00:00:00+02:002013-10-13T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2013-10-13:2013/Oct/13/linux-autumn-2013-recap/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/download-1-300x98.jpg" /></center></p>
<p>So this is it – Linux Autumn conference is over! We talked a lot, we hiked a beautiful mountain, we drank a loot of beer and meet new people. This year’s edition was placed in the Szczyrk – heart of Beskid Śląski and Żywiecki mountains – and weather was so great that i just couldn’t stay for all the talks inside the hotel and left for a while to take a hike to the biggest mountain in the hood (see images & videos that I shot on the end of this post).</p>
<p>The agenda was very interesting. I particularly liked:</p>
<ul>
<li>Pam_container – LXC + PAM + Python = ♥ (by Grzegorz Nosek). This topic is very close for me as I just falled in love with LXC.</li>
<li>Memory handling in Linux (by Maciej Nabożny). Performance topic are also very interesting for me – I spend a lot of time using strace / gdb on daily basis while developing some projects so it’s always nice to hear something new or to take a look from one’s perspective.</li>
<li>Smack – simple alternative to SELinux (by Rafał Krypa). I’m heavy SELinux user and also I just have to know about all the new security concepts. It was interesting but… I’ll stay with SELinux ;)</li>
<li>Know Your Tools – auditd (by Dariusz Puchalak). I use auditd on daily basis so just wanted to see how others do it. I can say – we all use it the way developer thought – so the proper way ;)</li>
<li>Take care of my logs (by Witold Duranek) – this was very interesting talk about marriage of Logstash, Elasticsearch, Kibana and RabbitMQ. Really great – something I will have to take a closer look & deploy.</li>
<li>Technomonopols (by Michał Woźniak) – this was flame & troll fight about out understaiding of the biggest social services monopolies. Quite interesting – and despite I hate troll fights & flame wars – this one draw my attention</li>
</ul>
<p>I also won the contest announced by guys from Samsung R&D Centre. There were two topics (improving Linux security and improving Linux boot time). I won Samsung Galaxy S3 thanks to my ideas about shortening the boot process time on Linux machines. I’ll write another post about that.</p>
<p>Unfortunately I didn’t make it to have a talk during this edition of Linux Autumn. Due to that I changed my worked recently I had too much things on my head and just weren’t able to prepare enough. I created 2 short lightning talks (about node.js security and managing resources with control groups) but unfortunately – there was no lightning talks event so I couldn’t give that presentations.</p>
<p>As I said – I love hiking the mountains so I took a trip to the summit of Skrzyczne mountain. You can view the photos below. I also created 2 time-lapse videos using my SGS3 – you can see dusk and dawn over this beautiful Skrzyczne mountain.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/YU6r3S1eFq0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe width="560" height="315" src="https://www.youtube.com/embed/xiaeb61nQFU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>Ganglia, multicast && KVM on CentOS2013-08-09T00:00:00+02:002013-08-09T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2013-08-09:2013/Aug/09/ganglia-multicast-kvm-on-centos/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/logo_small.jpg" /></center></p>
<p>This is just a short note – I have to post it as this problem was really annoying and I couldn’t find any solutions in Google, so had to resolve it by myself.</p>
<p>Don’t know what Ganglia is? Check here: http://ganglia.sourceforge.net/ – it just kicks ass :)</p>
<p>Problem? I installed gmond on all our hosts / guests (CentOS 5/6, KVM virt, latest Ganglia daemons), also configured properly gmetad daemons and started this whole stuff using multicast. And it was working – for a while. After about 10-20 minutes it just stopped working on KVM guests. I saw no charts for those machines – but gmonds (even in debug mode) didn’t reveal any problems. And KVM hosts’ charts were fine (mostly..).</p>
<p>One more thing – in KVM guests I always set “deaf = yes” (just don’t want to have too much multicast traffic – i set it to “no” only on some bare hosts).</p>
<p>Ok so the problem.. I hung for some time on tcpdump / strace and came to the root of this problem – somehow there was no multicast traffic on KVM guests (I turned off iptables on KVM guests for the time of this whole issue – resolving). After some time I found 2 possible root causes:</p>
<h4>1. On KVM hosts by default there is multicast filter set on: no-ip-multicast (You can check if You have it turned on with following command:</h4>
<div class="highlight"><pre><span></span>virsh<span class="w"> </span>nwfilter-list<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'no-ip-multicast'</span>
</pre></div>
<p>If it’s turned on – You can turn it off with:</p>
<div class="highlight"><pre><span></span>virsh<span class="w"> </span>nwfilter-undefine<span class="w"> </span>no-ip-multicast
service<span class="w"> </span>libvirtd<span class="w"> </span>restart
</pre></div>
<p>And that should do this part of the trick.</p>
<h4>2. And also – on CentOS KVM guests we have to turn off rp_filter in /etc/sysctl.conf:</h4>
<div class="highlight"><pre><span></span>net.ipv4.conf.default.rp_filter<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
</pre></div>
<p>and:</p>
<div class="highlight"><pre><span></span>sysctl<span class="w"> </span>-p
</pre></div>
<p>(You can try setting it to “loose mode” (so value: 2) instead of 0 – it can work for You and it’s always safer</p>
<p>Thats all for now. My sources for this one?</p>
<ul>
<li>Always in use: tcpdump / strace ;)</li>
<li>Oh my – KVM why do you do this for me (“for security You fool!”)? About - KVM firewall: http://libvirt.org/firewall.html</li>
<li>More about security and rp_filter: https://access.redhat.com/site/solutions/53031</li>
<li>And this nice RFC: http://www.ietf.org/rfc/rfc3704.txt</li>
</ul>MySQL HA2013-08-05T00:00:00+02:002013-08-05T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2013-08-05:2013/Aug/05/mysql-ha/<p><center><img alt="IMG" src="https://maciej.lasyk.info/images/mysql_logo-300x155.png" /></center></p>
<p>This is very interesting topic from sysops’ point of view. Honestly – It’s not that common to find a well organized HA MySQL environment (I’m talking here about good failover solution in master – slave scenario).</p>
<p>I won’t write here details for the most known solutions / scenarios – I think it’s enough to point those that one can check if it fits his needs.</p>
<ol>
<li>Master – master replication. In this scenario we don’t need automated failover – when we have some outage of one of those masters then it’s no problem as second one is still working. It’s good to use some LB layer between application and MySQL hosts that makes the app always hit the working server.</li>
<li>MySQL cluster. This is very comfortable scenario as MySQL cluster takes care of almost everything – this is “no single point of failure” solution. You can read about those here http://www.mysql.com/products/cluster/ and here http://dev.mysql.com/downloads/cluster/ –</li>
<li>MySQL Galera cluster – for synchronous, active – active multi – master topology: http://www.codership.com/content/using-galera-cluster</li>
<li>Failover – for now I know 2 solutions worth mentioning:<ol>
<li>Matsunobu Yoshinori created mysql-master-ha (working on MySQL servers 5.0, 5.1 and later) – this is very good tool, that handles automatic, manual and semi – manual failovers (even when slaves are in different relaylog positions). In addition You can use it when MySQL master host migration is needed. I recommend watching those slides: http://www.slideshare.net/matsunobu/automated-master-failover</li>
<li>MySQL utilities – this is quite simple daemon which primary job is to perform automated failover when needed. You can read about it here: http://www.clusterdb.com/mysql/mysql-utilities-webinar-qa-replay-now-available/</li>
</ol>
</li>
</ol>Linux for Game Development2013-06-24T00:00:00+02:002013-06-24T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2013-06-24:2013/Jun/24/linux-for-game-development/<p>Some time ago I wrote an article describing how gamedev looks like from Linux desktop perspective. It was published by <strong>The Linux Foundation</strong> on linux.com website. So it is the second time I landed on this great webpage - yikes!</p>
<p>You can read some of this story here: http://www.linux.com/news/featured-blogs/200-libby-clark/716918-linux-enterprise-user-stories-it-research-and-game-development – especially the part starting with “Linux for Game Development”.</p>Google Reader is dead. What now?2013-06-01T00:00:00+02:002013-06-01T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2013-06-01:2013/Jun/01/google-reader-is-dead-what-now/<p><center><img alt="img" src="https://maciej.lasyk.info/images/Screenshot-from-2013-07-01-112320-300x137.png" /></center></p>
<p>Ok, so Google Reader is dead. This is not the end of the world – everything one has to do is defeat his habits and find some other RSS solution. A couple of my friends asked me what kind of alternative I chose – and why not Feedly.</p>
<p>I answered shortly: I don’t like Feedly and I found something that assures me stable, secure and easy to migrate RSS solution. I decided to go with Tiny Tiny RSS – let’s Open Source ;)</p>
<p>But.. We have to host this solution on our own servers (ups). There is also Android App (ups – there’s none for iPhone). So what? We can ;) And this will be as secure as You deploy it – so roll the ball & remember that Your RSS feeds info is also interesting for someone looking at You ;)</p>
<p>There was some article on linux.com describing some installation details: http://www.linux.com/learn/tutorials/322446-weekend-project-replacing-google-reader-with-tiny-tiny-rss</p>DNS resolution problem – dig working, ping not2013-05-11T00:00:00+02:002013-05-11T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2013-05-11:2013/May/11/dns-resolution-problem-dig-working-ping-not/<p>Today I was reconfiguring my internal laptop network as I use virtual machines a lot (KVM ftw) and using <strong>/etc/hosts</strong> was not scaling anymore. I could use DNSmasq, but I prefer BIND – so I installed & chrooted it, and configured as caching-name server that properly resolve my internal zone ‘local’ (as I use addresses like ‘git.local’ or ‘dev1.local’ or whatever). Next I had to make NetworkManager to use this local DNS in the first place instead of those given from DHCP. I could of course edit properly <strong>/etc/resolv.conf</strong> and protect it with immutable attribute, but I suppose, that NM developers didn’t take it into consideration, that resolv.conf would be unwritable and hell knows what would happen then. So I added:</p>
<div class="highlight"><pre><span></span><span class="nv">DNS1</span><span class="o">=</span><span class="m">127</span>.0.0.1
<span class="nv">DNS2</span><span class="o">=</span><span class="m">8</span>.8.8.8
<span class="nv">DNS3</span><span class="o">=</span><span class="m">8</span>.8.4.4
</pre></div>
<p>To the <strong>/etc/sysconfig/network-scripts/ifcfg-Auto_WLANname</strong> and things were ok – after restarting network I had what I wanted to have in /etc/resolv.conf. I resolved VPN resolving problem similarly.</p>
<p>So now I wanted to start my work with VMs, and it appeared that I couldn’t make a connection to any of those:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-toshiba<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>ssh_git
ssh:<span class="w"> </span>Could<span class="w"> </span>not<span class="w"> </span>resolve<span class="w"> </span>hostname<span class="w"> </span>git.local:<span class="w"> </span>Name<span class="w"> </span>or<span class="w"> </span>service<span class="w"> </span>not<span class="w"> </span>known
</pre></div>
<p>Weird… And check this:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-toshiba<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>cat<span class="w"> </span>/etc/resolv.conf
<span class="c1"># Generated by NetworkManager</span>
search<span class="w"> </span><span class="nb">local</span>
nameserver<span class="w"> </span><span class="m">127</span>.0.0.1
nameserver<span class="w"> </span><span class="m">8</span>.8.8.8
nameserver<span class="w"> </span><span class="m">8</span>.8.4.4
</pre></div>
<p>So WTF with this DNS?</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-toshiba<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>+short<span class="w"> </span>git.local
<span class="m">192</span>.168.122.14
</pre></div>
<p>This time WTF was much bigger. Local DNS appears to be working correctly. So I thought that this Fedora claimed that won’t be resolving ‘local’ addresses via DNS. Just to confirm this idea I used tcpdump: <strong>tcpdump -n port 53</strong> – in the meantime trying to ping ‘git.local’ host. And nothing there – tcpdump was silent. bingo – Fedora was not using DNS at all to resolve this one. So why? Let’s see:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-toshiba<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>strace<span class="w"> </span>-e<span class="w"> </span>poll,select,connect,recvfrom,sendto<span class="w"> </span>ping<span class="w"> </span>git.local
connect<span class="o">(</span><span class="m">3</span>,<span class="w"> </span><span class="o">{</span><span class="nv">sa_family</span><span class="o">=</span>AF_FILE,<span class="w"> </span><span class="nv">sun_path</span><span class="o">=</span><span class="s2">"/var/run/nscd/socket"</span><span class="o">}</span>,<span class="w"> </span><span class="m">110</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>-1<span class="w"> </span>ENOENT<span class="w"> </span><span class="o">(</span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory<span class="o">)</span>
connect<span class="o">(</span><span class="m">3</span>,<span class="w"> </span><span class="o">{</span><span class="nv">sa_family</span><span class="o">=</span>AF_FILE,<span class="w"> </span><span class="nv">sun_path</span><span class="o">=</span><span class="s2">"/var/run/nscd/socket"</span><span class="o">}</span>,<span class="w"> </span><span class="m">110</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>-1<span class="w"> </span>ENOENT<span class="w"> </span><span class="o">(</span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory<span class="o">)</span>
connect<span class="o">(</span><span class="m">3</span>,<span class="w"> </span><span class="o">{</span><span class="nv">sa_family</span><span class="o">=</span>AF_FILE,<span class="w"> </span><span class="nv">sun_path</span><span class="o">=</span><span class="s2">"/var/run/avahi-daemon/socket"</span><span class="o">}</span>,<span class="w"> </span><span class="m">110</span><span class="o">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0</span>
ping:<span class="w"> </span>unknown<span class="w"> </span>host<span class="w"> </span>git.local
+++<span class="w"> </span>exited<span class="w"> </span>with<span class="w"> </span><span class="m">2</span><span class="w"> </span>+++
</pre></div>
<p>Ok – we have nscd in the first way (which is not running on my laptop) and next we have Avahi… but where the hell is DNS? Let’s see /etc/nsswitch.conf:</p>
<div class="highlight"><pre><span></span>hosts:<span class="w"> </span>files<span class="w"> </span>mdns4_minimal<span class="w"> </span><span class="o">[</span><span class="nv">NOTFOUND</span><span class="o">=</span><span class="k">return</span><span class="o">]</span><span class="w"> </span>dns<span class="w"> </span>myhostname
</pre></div>
<p>Ha – now everything is clear! You can read about Avahi, MDNS and ‘local’ domains here: http://avahi.org/wiki/AvahiAndUnicastDotLocal</p>
<p>Solution? There are two. Firstly we could just replace above nsswitch.conf entry with the following (of course only when NOT using Avahi):</p>
<div class="highlight"><pre><span></span>hosts:<span class="w"> </span>files<span class="w"> </span>dns<span class="w"> </span>myhostname
</pre></div>
<p>Second solution – we could reconfigure Avahi – just as You can read in the above URL:</p>
<div class="highlight"><pre><span></span><span class="c1">#/etc/avahi/avahi-daemon.conf</span>
<span class="o">[</span>server<span class="o">]</span>
domain-name<span class="o">=</span>.alocal
</pre></div>
<p>Now only restart Avahi, web browsers and everything should be working fine.</p>Using serial console to access virtual machines (Centos, KVM)2013-01-09T00:00:00+01:002013-01-09T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2013-01-09:2013/Jan/09/using-serial-console-to-access-virtual-machines-centos-kvm/<p>Last time I promised, that I’ll write how to connect to VM via serial console. So – this time very shortly and without any explanations:</p>
<p>On the guest OS:</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span>ttyS0<span class="w"> </span>>><span class="w"> </span>/etc/securetty
</pre></div>
<p>Now add following param to kernel params in <strong>/etc/grub.conf</strong>:</p>
<div class="highlight"><pre><span></span><span class="nv">console</span><span class="o">=</span>ttyS0
</pre></div>
<p>And the last thing:</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span><span class="s2">"S0:12345:respawn:/sbin/agetty ttyS0 115200"</span><span class="w"> </span>>><span class="w"> </span>/etc/inittab<span class="w"> </span><span class="o">&&</span><span class="w"> </span>init<span class="w"> </span>q
</pre></div>
<p>Now restart the guest OS, and You’re good to connect to VM via:</p>
<p><strong>virsh console guest_id</strong></p>
<p>If You feel like interested in what happened above – please ask questions in comments – I’ll explain :)</p>Installing KVM guest OS via VNC over SSH tunnel2013-01-03T00:00:00+01:002013-01-03T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2013-01-03:2013/Jan/03/installing-kvm-guest-os-via-vnc-over-ssh-tunnel/<p>Verry merry and Christmas is over. And I’ve bought myself a brand-new server for this occasion :D So this is it – enough of doing-nothing or not-doing-anything and I had to start migrating services from the old box. And as old machine is just bare-metal env, where users kill each other for memory, than I decided – no more. KVM, cgroups and hell with ya guys – You won’t ever know about each other! And all this for the same price (as the old box has 2 years and I paid the same price for the new one, where I’ve got 8x more RAM, 2x more storage and some quad-core…).</p>
<p>Ok enough of this talking. So I’ve got clean CentOS 6.3 installation with basic KVM environment and SELinux set to Permissive mode (You could leave it in Enforcing, having to “<strong>chcon –reference /var/lib/libvirt/images /your/vm/repodir</strong>” – but i see no point in using SELinux in host OS – this would eat too many resources, and is at all not needed – what you have to do on host OS for security is using very strict rules.</p>
<p>So… Centos, KVM, Permissive and We’re ready to engage. For lazy guys I suggest using Virtual Manger (virt-manager) where You can click through the whole guest configuration process (for making this work You should turn off iptables for a while or open some virt-manager TCP ports). But as virt-manager is for lame, then we write on the terminal:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@cubryna<span class="w"> </span>iso<span class="o">]</span><span class="c1"># virt-install -r 2048 --accelerate -n VM-docent --disk path=/vm/VM-docent/VM-docent.img,size=50 --cdrom CentOS-6.3-x86_64-minimal.iso --vcpus=2 --vnc --os-type linux --hvm --vncport=65322</span>
Starting<span class="w"> </span>install...
Creating<span class="w"> </span>storage<span class="w"> </span>file<span class="w"> </span>VM-docent.img<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">50</span><span class="w"> </span>GB<span class="w"> </span><span class="m">00</span>:00
Creating<span class="w"> </span>domain...<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">0</span><span class="w"> </span>B<span class="w"> </span><span class="m">00</span>:00
Cannot<span class="w"> </span>open<span class="w"> </span>display:
Run<span class="w"> </span><span class="s1">'virt-viewer --help'</span><span class="w"> </span>to<span class="w"> </span>see<span class="w"> </span>a<span class="w"> </span>full<span class="w"> </span>list<span class="w"> </span>of<span class="w"> </span>available<span class="w"> </span><span class="nb">command</span><span class="w"> </span>line<span class="w"> </span>options
Domain<span class="w"> </span>installation<span class="w"> </span>still<span class="w"> </span><span class="k">in</span><span class="w"> </span>progress.<span class="w"> </span>You<span class="w"> </span>can<span class="w"> </span>reconnect<span class="w"> </span>to
the<span class="w"> </span>console<span class="w"> </span>to<span class="w"> </span><span class="nb">complete</span><span class="w"> </span>the<span class="w"> </span>installation<span class="w"> </span>process.
</pre></div>
<p>And installation is running. Now We’d like to connect to it – so VNC FTW! But…</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@cubryna<span class="w"> </span>iso<span class="o">]</span><span class="c1"># netstat -nlp | grep 65322</span>
tcp<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">127</span>.0.0.1:65322<span class="w"> </span><span class="m">0</span>.0.0.0:*<span class="w"> </span>LISTEN<span class="w"> </span><span class="m">12768</span>/qemu-kvm
</pre></div>
<p>VNC daemon is safely listening only on localhost, so We have to try some different way. We could make this daemon to listen also on WAN interface, but this would be to lame and risky. So we create SSH tunnel:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-toshiba<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>ssh<span class="w"> </span>-p<span class="w"> </span><span class="m">65234</span><span class="w"> </span>docent@cubryna.makaronzserem.eu<span class="w"> </span>-L<span class="w"> </span><span class="m">65322</span>:127.0.0.1:65322
RSA<span class="w"> </span>host<span class="w"> </span>key<span class="w"> </span><span class="k">for</span><span class="w"> </span>IP<span class="w"> </span>address<span class="w"> </span><span class="s1">'5.135.178.98'</span><span class="w"> </span>not<span class="w"> </span><span class="k">in</span><span class="w"> </span>list<span class="w"> </span>of<span class="w"> </span>known<span class="w"> </span>hosts.
Last<span class="w"> </span>login:<span class="w"> </span>Thu<span class="w"> </span>Jan<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">20</span>:26:48<span class="w"> </span><span class="m">2013</span><span class="w"> </span>from<span class="w"> </span>neostradaaa
CentOS<span class="w"> </span>release<span class="w"> </span><span class="m">6</span>.3<span class="w"> </span><span class="o">(</span>Final<span class="o">)</span>
Linux<span class="w"> </span>cubryna.makaronzserem.eu<span class="w"> </span><span class="m">2</span>.6.32-279.19.1.el6.x86_64<span class="w"> </span><span class="c1">#1 SMP Wed Dec 19 07:05:20 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux</span>
ip<span class="w"> </span>:<span class="w"> </span><span class="m">5</span>.135.178.98
hostname<span class="w"> </span>:<span class="w"> </span>cubryna.makaronzserem.eu
<span class="o">[</span>docent@cubryna<span class="w"> </span>~<span class="o">]</span>$
</pre></div>
<p>And just after that We can safely connect our VNC client (like Tiger-VNC) to our installation process using host 127.0.0.1:65322 :)</p>
<p>Of course after successful installation, I suggest creating clone of our brand-new VPS just to save some time for future installations – use virt-clone command to do this.</p>
<p>And that’s all for now – next time I’ll write how to configure guest OS to make it possible to use virsh console command to connect to it via serial console.</p>How to change OVH kernel to normal Centos kernel2012-12-23T00:00:00+01:002012-12-23T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2012-12-23:2012/Dec/23/how-to-change-ovh-kernel-to-normal-centos-kernel/<p>I’ve just installed new OVH server on Centos 6.3. And I spotted a problem – I couldn’t install KVM services due to lack of KVM kernel modules. And I wasn’t able to load them because this fresh installation used some OVH kernel which was wiped out of those modules…</p>
<p>So I had two ways of solving this issue. Recompiling kernel from scratch and adding KVM modules (long way – and what for? It just consumes so much time…) or.. just revert stock Centos kernel :) And this was the way I chose.</p>
<p>Whole procedure is just trivial. First thing – let’s install new kernel (stock one) with yum:</p>
<div class="highlight"><pre><span></span>root@ks3283784<span class="w"> </span>~<span class="o">]</span><span class="c1"># yum install kernel</span>
Loaded<span class="w"> </span>plugins:<span class="w"> </span>fastestmirror
Loading<span class="w"> </span>mirror<span class="w"> </span>speeds<span class="w"> </span>from<span class="w"> </span>cached<span class="w"> </span>hostfile
<span class="w"> </span>*<span class="w"> </span>base:<span class="w"> </span>mirror.ovh.net
<span class="w"> </span>*<span class="w"> </span>extras:<span class="w"> </span>mirror.ovh.net
<span class="w"> </span>*<span class="w"> </span>updates:<span class="w"> </span>mirror.ovh.net
base<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">3</span>.7<span class="w"> </span>kB<span class="w"> </span><span class="m">00</span>:00
extras<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">3</span>.5<span class="w"> </span>kB<span class="w"> </span><span class="m">00</span>:00
updates<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">3</span>.5<span class="w"> </span>kB<span class="w"> </span><span class="m">00</span>:00
Setting<span class="w"> </span>up<span class="w"> </span>Install<span class="w"> </span>Process
Package<span class="w"> </span>kernel-2.6.32-279.19.1.el6.x86_64<span class="w"> </span>already<span class="w"> </span>installed<span class="w"> </span>and<span class="w"> </span>latest<span class="w"> </span>version
Nothing<span class="w"> </span>to<span class="w"> </span><span class="k">do</span>
</pre></div>
<p>When installed, we need to include it in the boot loader – which is Grub by default in Centos. In order to do it properly, we have to know our kernel version (so this one we’re given after <strong>yum install kernel</strong> command, so: <strong>kernel-2.6.32-279.19.1.el6.x86_64</strong> in my case and the device, which hosts our root partition – we can check this with mount command and the contents of <strong>/etc/fstab</strong> directory:</p>
<div class="highlight"><pre><span></span>/dev/md1<span class="w"> </span>/<span class="w"> </span>ext4<span class="w"> </span><span class="nv">errors</span><span class="o">=</span>remount-ro<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span>
/dev/md2<span class="w"> </span>/home<span class="w"> </span>ext4<span class="w"> </span>defaults<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">2</span>
/dev/sda3<span class="w"> </span>none<span class="w"> </span>swap<span class="w"> </span>defaults<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
/dev/sdb3<span class="w"> </span>none<span class="w"> </span>swap<span class="w"> </span>defaults<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span>
</pre></div>
<p>Above we see that / directory is hosted on /dev/md1 device. So now we can modify <strong>/boot/grub/grub.conf</strong>:</p>
<div class="highlight"><pre><span></span><span class="nv">default</span><span class="o">=</span><span class="m">0</span>
<span class="nv">timeout</span><span class="o">=</span><span class="m">5</span>
<span class="w"> </span>title<span class="w"> </span>linux<span class="w"> </span>centos6_64
<span class="w"> </span>root<span class="w"> </span><span class="o">(</span>hd0,0<span class="o">)</span>
<span class="w"> </span>kernel<span class="w"> </span>/boot/vmlinuz-2.6.32-279.19.1.el6.x86_64<span class="w"> </span><span class="nv">root</span><span class="o">=</span>/dev/md1/strong<span class="p">&</span>gt<span class="p">;</span>
<span class="w"> </span>initrd<span class="w"> </span>/boot/initramfs-2.6.32-279.19.1.el6.x86_64.img
</pre></div>
<p>Now after double-check we just reboot our system and wait.. If it’s up then let’s be sure, that we got our stock centos kernel:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@ks3283784<span class="w"> </span>~<span class="o">]</span><span class="c1"># uname -r</span>
<span class="m">2</span>.6.32-279.19.1.el6.x86_64
</pre></div>
<p>So we’re good :) If for some reason server is not coming back after reboot then We can rescue it by booting it into rescue-mode in OVH manager. Then we need to replace <strong>/boot/grub/grub.conf</strong> with our backup <strong>/boot/grub/grub.conf.bak</strong></p>Safe WordPress management via SSL / HTTPS2012-09-17T00:00:00+02:002012-09-17T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2012-09-17:2012/Sep/17/safe-wordpress-management-via-ssl-https/<p>Why would you like to secure your <strong>wp-admin</strong> session with SSL? Remember – Big Brother is always watching – so don’t make his life easy.</p>
<p>In order to use SSL in wp-admin the most important thing is to enable SSL in WWW server’s vhost (eg. Apache). When SSL is turned on for your WordPress domain it will work just out of the box.</p>
<p>So what is also important here? You should always make users use SSL in wp-admin sessions. So make it obligatory. You can do it using mod_rewrite in Apache webserver (httpd.conf or .htaccess):</p>
<div class="highlight"><pre><span></span>RewriteEngine<span class="w"> </span>On
RewriteBase<span class="w"> </span>/
RewriteCond<span class="w"> </span>%<span class="o">{</span>HTTPS<span class="o">}</span><span class="w"> </span>!<span class="o">=</span>on
RewriteRule<span class="w"> </span>“^<span class="o">(</span>/wp-admin/.*<span class="o">)</span>”<span class="w"> </span>“https://%<span class="o">{</span>HTTP_HOST<span class="o">}</span><span class="nv">$1</span>”<span class="w"> </span><span class="o">[</span><span class="nv">R</span><span class="o">=</span><span class="m">301</span>,L<span class="o">]</span>
</pre></div>
<p>Or simplier - editing Your <strong>wp-config.php</strong> file - add below line:</p>
<div class="highlight"><pre><span></span>define<span class="o">(</span>‘FORCE_SSL_ADMIN’,<span class="w"> </span><span class="nb">true</span><span class="o">)</span><span class="p">;</span>
</pre></div>
<p>somewhere before the folliwing line:</p>
<div class="highlight"><pre><span></span>require_once<span class="o">(</span>ABSPATH<span class="w"> </span>.<span class="w"> </span>‘wp-settings.php’<span class="o">)</span><span class="p">;</span>
</pre></div>
<p>And that should do the trick!</p>Short brief from LinuxCon Europe 2012 / Barcelona2012-09-14T00:00:00+02:002012-09-14T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2012-09-14:2012/Sep/14/short-brief-from-linuxcon-europe-2012-barcelona/<p><strong>LinuxCon Europe 2012</strong> is a wrap! And it was really brainfuck storm on the space level, that my brain said loudly: “no more!”, I absorbed so much coffee, that my magnesium level shrank to some negative number.. and what for? Two things happened which are worth of saying loudly or rather writing on this blog.</p>
<p>First one – after so many years I successfully have a photo with Linus Torvalds! I also talked to him a little (very..) and he shook my hand (which I will not wash for weeks now ;) ). Foto below, on the right side is my friend from work, CTO at Ganymede:</p>
<p><center><img alt="img" src="https://maciej.lasyk.info/images/600565_10151122270180911_20684212_n-300x225.jpg" /></center></p>
<p>Second thing is - what Bill Kent from <strong>The Linux Foundation</strong> wrote me after I tried myself at the Linux Trivia contest:</p>
<blockquote>
<p>You were <strong>one</strong> of only <strong>four</strong> people who answered <strong>every</strong> question <strong>correctly</strong>.
I'm happy to inform you that you received our second prize - the free
Linux soccer/football jersey!</p>
</blockquote>
<p><center><img alt="img" src="https://maciej.lasyk.info/images/403558_10151123392185911_806222762_n-300x225.jpg" /></center></p>
<p>As You see - I'm in the Linux football team! (and don’t use this word: "soccer" You Americans - football was used first in old continent – take this "soccer" thing or use "rugby" instead :D</p>
<p>I must also write, that above picture was placed at linux.com webpage! <a href="http://www.linux.com/news/galleries/linuxcon-europe-2012-day-3-linus-speaks/this-years-speaker-gift">http://www.linux.com/news/galleries/linuxcon-europe-2012-day-3-linus-speaks/this-years-speaker-gift</a></p>
<p>For now that’s all – I’ll try to put some more words about LinuxCon – but honestly I don’t know when I will have time to unzip && untar all the brainfuck notes and put them all together in some reasonable form.. ;)</p>Lighttpd expire.url mixed entries2012-07-05T00:00:00+02:002012-07-05T00:00:00+02:00Maciej Lasyktag:maciej.lasyk.info,2012-07-05:2012/Jul/05/lighttpd-expireurl-mixed-entries/<p>Lately I had to set expiration settings for some webcontent in whole virtual host in Lighttpd. So i set:</p>
<div class="highlight"><pre><span></span>expire.url<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">(</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">=</span><span class="p">&</span>><span class="w"> </span><span class="s2">"access 3 months"</span><span class="w"> </span><span class="o">)</span>
</pre></div>
<p>And tested if it’s working fine:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@centos62-minimal<span class="w"> </span>~<span class="o">]</span><span class="c1"># wget -S --header="Host: test" "http://192.168.122.245:85/test.css" --2012-07-05 16:32:05-- http://192.168.122.245:85/test.css Connecting to 192.168.122.245:85... connected. HTTP request sent, awaiting response... HTTP/1.0 200 OK Expires: Mon, 03 Oct 2012 14:32:05 GMT Cache-Control: max-age=5184000 Content-Type: text/css Accept-Ranges: bytes ETag: "1677" Last-Modified: Thu, 05 Jul 2012 14:24:50 GMT Content-Length: 5 Connection: close Date: Thu, 05 Jul 2012 14:32:05 GMT Server: lighttpd/1.4.30 Length: 5 [text/css]</span>
</pre></div>
<p>So it looks fine (expiration is 3 months in future). Now I wanted to change expiration setting for a particular subdirectory <strong>dirtest</strong>:</p>
<div class="highlight"><pre><span></span>expire.url<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">(</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">=</span><span class="p">&</span>><span class="w"> </span><span class="s2">"access 3 months"</span><span class="w"> </span>,<span class="w"> </span><span class="s2">"/dirtest/"</span><span class="w"> </span><span class="o">=</span><span class="p">&</span>><span class="w"> </span><span class="s2">"access 5 minutes"</span><span class="w"> </span><span class="o">)</span>
</pre></div>
<p>And I found it’s not working:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@centos62-minimal<span class="w"> </span>~<span class="o">]</span><span class="c1"># wget -S --header="Host: test" "http://192.168.122.245:85/dirtest/test.js" --2012-07-05 16:32:05-- http://192.168.122.245:85/dirtest/test.js Connecting to 192.168.122.245:85... connected. HTTP request sent, awaiting response... HTTP/1.0 200 OK Expires: Mon, 03 Oct 2012 14:32:05 GMT Cache-Control: max-age=5184000 Content-Type: text/css Accept-Ranges: bytes ETag: "1677" Last-Modified: Thu, 05 Jul 2012 14:24:50 GMT Content-Length: 5 Connection: close Date: Thu, 05 Jul 2012 14:32:05 GMT Server: lighttpd/1.4.30 Length: 5 [text/css]</span>
</pre></div>
<p>So WTF? It appears that it makes a difference in what order you’ve put the entries in the conf. This one is working fine:</p>
<div class="highlight"><pre><span></span>expire.url<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">(</span><span class="w"> </span><span class="s2">"/dirtest/"</span><span class="w"> </span><span class="o">=</span><span class="p">&</span>><span class="w"> </span><span class="s2">"access 5 minutes"</span><span class="w"> </span>,<span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="o">=</span><span class="p">&</span>><span class="w"> </span><span class="s2">"access 3 months"</span><span class="w"> </span><span class="o">)</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="o">[</span>root@centos62-minimal<span class="w"> </span>~<span class="o">]</span><span class="c1"># wget -S --header="Host: test" "http://192.168.122.245:85/dirtest/test.js" --2012-07-05 16:32:09-- http://192.168.122.245:85/dirtest/test.js" Connecting to 192.168.122.245:85... connected. HTTP request sent, awaiting response... HTTP/1.0 200 OK Expires: Thu, 05 Jul 2012 14:34:09 GMT Cache-Control: max-age=120 Content-Type: text/javascript Accept-Ranges: bytes ETag: "3840495965" Last-Modified: Thu, 05 Jul 2012 14:27:37 GMT Content-Length: 3 Connection: close Date: Thu, 05 Jul 2012 14:32:09 GMT Server: lighttpd/1.4.30 Length: 3 [text/javascript]</span>
</pre></div>
<p>So just remember about proper ordering of items in expire.url entry :)</p>
<p>(tested on <strong>lighttpd-1.4.30</strong>)</p>Dou you kanban?2012-03-04T00:00:00+01:002012-03-04T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2012-03-04:2012/Mar/04/dou-you-kanban/<h1>Introduction to Lean Management with Kanban in IT</h1>
<p>Recently I gave a speech about using Kanban in Operations management process. I prepared it the night before (It was like an emergency – I had to fill the 60min gap in the 2-hours speech block), so didn’t have too much time (to be honest – I had only one night). I decided to prepare it with prezi.com presentation engine – I’ve seen it before and always wanted to go this unusual way. But I had a few troubles with this engine:</p>
<ul>
<li>I decided to prepare this “prezi” in polish language (right now I’m translating it into english). I could not use polish fonts under my Fedora 15 / Chrome – very weird. Normally I have no problems with PL fonts in Flash apps… and of course It was late night, so I haven’t time to work on this problem</li>
<li>I wanted to download the offline presentation to be able to show it independently of internet connection. I was able to export it only as an exe file.. the only way to play it on my Fedora was to use Wine – It was ok until I tried to go fullscreen – app broken everytime (in all my 3 boxes I tried on – all on Fedoras)</li>
<li>Finally, the speech was ok as I know from the participants, but one of the technical problems was a low framerate during slides transformations – and this is what the Prezi is all about (finally I played it with online viewer in prezi.com). I used Aces Extensa 5220 with 2GB RAM, Celeron 1.73 GHz and Intel GMA X3100</li>
</ul>
<p>Despite all this issues I will try to improve next time – I’ll try to solve the PL fonts problem and I’ll also investigate the low performance problem. Below I’ve embedded the PL version. When I’ve finished the EN version I will add it below:</p>
<iframe src="https://prezi.com/embed/xncyx7ux34ll/" id="iframe_container" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen="" allow="autoplay; fullscreen" height="315" width="560"></iframe>ssh_exchange_identification: Connection closed by remote host2012-01-30T00:00:00+01:002012-01-30T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2012-01-30:2012/Jan/30/ssh_exchange_identification-connection-closed-by-remote-host/<p>Recently trying to connect via <strong>ssh</strong> using <strong>cssh</strong> to a particular server from 26 other boxes at the same time I wasn’t able to connect from some of those boxes and saw this message on those:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@machine<span class="w"> </span>.ssh<span class="o">]</span><span class="c1"># ssh -p 56789 user@othermachine.com</span>
ssh_exchange_identification:<span class="w"> </span>Connection<span class="w"> </span>closed<span class="w"> </span>by<span class="w"> </span>remote<span class="w"> </span>host
</pre></div>
<p>This was due to server configuration. Maximum default number of simultaneous connections tries (login attemps) is defined in <strong>/etc/ssh/sshd_config</strong>:</p>
<div class="highlight"><pre><span></span>MaxStartups<span class="w"> </span><span class="m">10</span>
</pre></div>
<p>We can change above value or use new-style format:</p>
<div class="highlight"><pre><span></span>MaxStartups<span class="w"> </span><span class="m">10</span>:30:60
</pre></div>
<p>Which stands for:</p>
<ul>
<li>10 – number of allowed simultaneous connections attempts. Above this number SSHD will start to randomly drop connections with percentage chance of 30%</li>
<li>60 - number of simultaneous connections attempts after which SSHD will drop every new connection</li>
</ul>Digging know how (howto)2012-01-13T00:00:00+01:002012-01-13T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2012-01-13:2012/Jan/13/digging-know-how-howto/<p><strong>Dig</strong> is one of the most important tools that every sysop uses in day-to-day work. It gives us the possibility to trace resolving path of domains, checking status of domain records or even getting whole definitions of records for a particular domain (under some circumstances…). We won’t write here an essay about DNS and it’s functionality – You san always read it here: http://en.wikipedia.org/wiki/Domain_Name_System or even better, here: https://webhostinggeeks.com/guides/dns/</p>
<p>Assuming, that You ave already dig installed (if not - try <strong>yum install bind-utils</strong> on CentOs or whatever else on different distros) We can start with explaining how to use properly dig.</p>
<h3>1. Simple query</h3>
<p>Let’s check Gamedesire’s domain <strong>www.gamedesire.com</strong>:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>ganymede<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>sysop.gamedesire.com
<span class="p">;</span><span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>DiG<span class="w"> </span><span class="m">9</span>.8.1-P1-RedHat-9.8.1-3.P1.fc15<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>sysop.gamedesire.com
<span class="p">;;</span><span class="w"> </span>global<span class="w"> </span>options:<span class="w"> </span>+cmd
<span class="p">;;</span><span class="w"> </span>Got<span class="w"> </span>answer:
<span class="p">;;</span><span class="w"> </span>-<span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span>HEADER<span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;</span>
</pre></div>
<p>Above We can see a bunch of data.. “status: NOERROR” is answer status – NOERROR means, that query was resolved properly. ANSWER SECTION contains of information about resolved records (here We can see 2 A records for www.gamedesire.com: 174.123.95.100 and 174.123.95.101). Next We have AUTHORITY SECTION which gives us an answer to the question: “what are the authoritative DNS servers for this domain?”. We have also some statistic data (query time, ADDITIONAL SECTION with DNS servers info etc). And remember to check answer flags (in above example: qr rd ra). You can find explanation of those in RFC 1035 §4.1.1..</p>
<p>What happens when domain name is not resolved properly? Let’s try with non-existing sysop.gamedesire.com:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>ganymede<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>sysop.gamedesire.com
<span class="p">;</span><span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>DiG<span class="w"> </span><span class="m">9</span>.8.1-P1-RedHat-9.8.1-3.P1.fc15<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>sysop.gamedesire.com
<span class="p">;;</span><span class="w"> </span>global<span class="w"> </span>options:<span class="w"> </span>+cmd
<span class="p">;;</span><span class="w"> </span>Got<span class="w"> </span>answer:
<span class="p">;;</span><span class="w"> </span>-<span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span>HEADER<span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;</span>
</pre></div>
<p>As we can see the answer status is NXDOMAIN here – it means NoneXistingDomain.</p>
<h3>2. Querying specific records</h3>
<p>There are plenty of DNS record types. Most commonly used are A, MX, TXT and CNAME. Here is the explanation and complete list of those types: http://en.wikipedia.org/wiki/List_of_DNS_record_types. So now – how can We query for a MX domain?</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>ganymede<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>mx<span class="w"> </span>google.com
<span class="p">;</span><span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>DiG<span class="w"> </span><span class="m">9</span>.8.1-P1-RedHat-9.8.1-3.P1.fc15<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>mx<span class="w"> </span>google.com
<span class="p">;;</span><span class="w"> </span>global<span class="w"> </span>options:<span class="w"> </span>+cmd
<span class="p">;;</span><span class="w"> </span>Got<span class="w"> </span>answer:
<span class="p">;;</span><span class="w"> </span>-<span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span>HEADER<span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;</span>
</pre></div>
<p>So We see above in ANSWER SECTION all found MX records. We can ask for any record type with this method.</p>
<h3>3. Tracing DNS query</h3>
<p>Every DNS query takes some hierarchical steps. Knowing those could be very helpful under some circumstances:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>ganymede<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>+trace<span class="w"> </span>www.wikipedia.org
<span class="p">;</span><span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>DiG<span class="w"> </span><span class="m">9</span>.8.1-P1-RedHat-9.8.1-3.P1.fc15<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>+trace<span class="w"> </span>www.wikipedia.org
<span class="p">;;</span><span class="w"> </span>global<span class="w"> </span>options:<span class="w"> </span>+cmd
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>k.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>m.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>j.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>l.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>b.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>f.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>h.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>g.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>c.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>e.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>d.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>i.root-servers.net.
.<span class="w"> </span><span class="m">209351</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>a.root-servers.net.
<span class="p">;;</span><span class="w"> </span>Received<span class="w"> </span><span class="m">512</span><span class="w"> </span>bytes<span class="w"> </span>from<span class="w"> </span><span class="m">192</span>.168.1.1#53<span class="o">(</span><span class="m">192</span>.168.1.1<span class="o">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">693</span><span class="w"> </span>ms
org.<span class="w"> </span><span class="m">172800</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>b2.org.afilias-nst.org.
org.<span class="w"> </span><span class="m">172800</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>a0.org.afilias-nst.info.
org.<span class="w"> </span><span class="m">172800</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>a2.org.afilias-nst.info.
org.<span class="w"> </span><span class="m">172800</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>c0.org.afilias-nst.info.
org.<span class="w"> </span><span class="m">172800</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>b0.org.afilias-nst.org.
org.<span class="w"> </span><span class="m">172800</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>d0.org.afilias-nst.org.
<span class="p">;;</span><span class="w"> </span>Received<span class="w"> </span><span class="m">437</span><span class="w"> </span>bytes<span class="w"> </span>from<span class="w"> </span><span class="m">128</span>.8.10.90#53<span class="o">(</span><span class="m">128</span>.8.10.90<span class="o">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">379</span><span class="w"> </span>ms
wikipedia.org.<span class="w"> </span><span class="m">86400</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>ns1.wikimedia.org.
wikipedia.org.<span class="w"> </span><span class="m">86400</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>ns0.wikimedia.org.
wikipedia.org.<span class="w"> </span><span class="m">86400</span><span class="w"> </span>IN<span class="w"> </span>NS<span class="w"> </span>ns2.wikimedia.org.
<span class="p">;;</span><span class="w"> </span>Received<span class="w"> </span><span class="m">147</span><span class="w"> </span>bytes<span class="w"> </span>from<span class="w"> </span><span class="m">199</span>.19.54.1#53<span class="o">(</span><span class="m">199</span>.19.54.1<span class="o">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">332</span><span class="w"> </span>ms
www.wikipedia.org.<span class="w"> </span><span class="m">3600</span><span class="w"> </span>IN<span class="w"> </span>CNAME<span class="w"> </span>wikipedia-lb.wikimedia.org.
wikipedia-lb.wikimedia.org.<span class="w"> </span><span class="m">600</span><span class="w"> </span>IN<span class="w"> </span>CNAME<span class="w"> </span>wikipedia-lb.esams.wikimedia.org.
wikipedia-lb.esams.wikimedia.org.<span class="w"> </span><span class="m">3600</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">91</span>.198.174.225
<span class="p">;;</span><span class="w"> </span>Received<span class="w"> </span><span class="m">121</span><span class="w"> </span>bytes<span class="w"> </span>from<span class="w"> </span><span class="m">208</span>.80.152.142#53<span class="o">(</span><span class="m">208</span>.80.152.142<span class="o">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">157</span><span class="w"> </span>ms
</pre></div>
<p>We see above all the DNS query steps – first question to the root servers for a proper TLD DNS server (org), next the question to the TLD org’s server for a proper wikipedia.org NS servers, and then the question to the wikipedia’s NS servers for a resolution to the name ‘wikipedia.org’ which appears to be a CNAMEs for something more…</p>
<h3>4. Shortening the output</h3>
<p>Digging is a quite verbose action – by design – much verbosity is good for debugging purposes. It’s good to know how can We reduce output of this command – for example when We would like to wrap dig command with some monitoring script. For this purpose I suggest to get known with:</p>
<div class="highlight"><pre><span></span>+nostats
+nocmd
+noquestion
+short
</pre></div>
<p>Let’s try with the strongest one from above - <strong>+short</strong> will reduce all the dig’s “noise”:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>ganymede<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>+short<span class="w"> </span>www.gamedesire.com
<span class="m">174</span>.123.95.101
<span class="m">174</span>.123.95.100
</pre></div>
<p>We can join <strong>+short</strong> with eg. “+trace”:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>ganymede<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>+short<span class="w"> </span>+trace<span class="w"> </span>www.gamedesire.com
NS<span class="w"> </span>l.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>c.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>g.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>m.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>b.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>a.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>i.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>j.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>d.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>e.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>h.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>f.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
NS<span class="w"> </span>k.root-servers.net.<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">192</span>.168.1.1<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">33</span><span class="w"> </span>ms.
A<span class="w"> </span><span class="m">174</span>.123.95.100<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">207</span>.218.247.135<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">173</span><span class="w"> </span>ms.
A<span class="w"> </span><span class="m">174</span>.123.95.101<span class="w"> </span>from<span class="w"> </span>server<span class="w"> </span><span class="m">207</span>.218.247.135<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">173</span><span class="w"> </span>ms.
</pre></div>
<p>You can try yourself with other params.</p>
<h3>5. Asking specific DNS server</h3>
<p>It is very good practice during checking DNS resolutions (especially while transferring domains etc) to ask query to a couple of DNS servers. We can ask particular DNS server (but only when this server allows us to do so). Remember that by default dig uses DNSes listed in Your /etc/resolv.conf file. Let’s try to ask google’s DNSes first:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>ganymede<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>+short<span class="w"> </span>@8.8.8.8<span class="w"> </span>www.gamedesire.com
<span class="m">174</span>.123.95.101
<span class="m">174</span>.123.95.100
</pre></div>
<p>And now some ThePlanet's:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>ganymede<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>+short<span class="w"> </span>@ns1.theplanet.com<span class="w"> </span>www.gamedesire.com
<span class="m">174</span>.123.95.101
<span class="m">174</span>.123.95.100
</pre></div>
<h3>6. The authority</h3>
<p>In this example:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>www.gamedesire.com<span class="w"> </span>@ns1.theplanet.com
<span class="p">;</span><span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>DiG<span class="w"> </span><span class="m">9</span>.8.1-P1-RedHat-9.8.1-3.P1.fc15<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>www.gamedesire.com<span class="w"> </span>@ns1.theplanet.com
<span class="p">;;</span><span class="w"> </span>global<span class="w"> </span>options:<span class="w"> </span>+cmd
<span class="p">;;</span><span class="w"> </span>Got<span class="w"> </span>answer:
<span class="p">;;</span><span class="w"> </span>-<span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span>HEADER<span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;</span>
</pre></div>
<p>We can see, that AA bit is set here (flags: qr aa rd). So ns1.theplanet.com is authoritative for gamedesire.com domain (as in AUTHORITY section). Now let’s try to dig it again using some other DNS server:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>dig<span class="w"> </span>www.gamedesire.com
<span class="p">;</span><span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>DiG<span class="w"> </span><span class="m">9</span>.8.1-P1-RedHat-9.8.1-3.P1.fc15<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>lt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>www.gamedesire.com
<span class="p">;;</span><span class="w"> </span>global<span class="w"> </span>options:<span class="w"> </span>+cmd
<span class="p">;;</span><span class="w"> </span>Got<span class="w"> </span>answer:
<span class="p">;;</span><span class="w"> </span>-<span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;&</span>amp<span class="p">;</span>gt<span class="p">;</span>HEADER<span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;</span>
</pre></div>
<p>Here We can see, that there is no AA bit set, and also TTL values are lower than 86400 for www.gamedesire.com (We can see 19128 value). Why is that? Because this answer is cached somewhere in the middle and our current DNS server is not authorative for gamedesire.com . If We would repeat this query this TTL value would be dropping every each question. We can’t tell looking at above dig output where this query was cached – to know this We would have to repeat the query with recursion disabled and step manually through all the DNS tree (but in 9/10 cases It will be You local DNS cache… which is caching answers).</p>
<h3>7. Tracing DIG execution</h3>
<p>As I wrote above We can set <strong>+trace</strong> param using dig to trace resolving path in the DNS tree. But how can We trace what exact queries are sent and received? Surely with tcpdump:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>root@docent-desktop<span class="w"> </span>~<span class="o">]</span><span class="c1"># tcpdump -i p2p1 -s1024 udp port domain</span>
tcpdump:<span class="w"> </span>verbose<span class="w"> </span>output<span class="w"> </span>suppressed,<span class="w"> </span>use<span class="w"> </span>-v<span class="w"> </span>or<span class="w"> </span>-vv<span class="w"> </span><span class="k">for</span><span class="w"> </span>full<span class="w"> </span>protocol<span class="w"> </span>decode
listening<span class="w"> </span>on<span class="w"> </span>p2p1,<span class="w"> </span>link-type<span class="w"> </span>EN10MB<span class="w"> </span><span class="o">(</span>Ethernet<span class="o">)</span>,<span class="w"> </span>capture<span class="w"> </span>size<span class="w"> </span><span class="m">512</span><span class="w"> </span>bytes
<span class="m">10</span>:54:17.047260<span class="w"> </span>IP<span class="w"> </span>pma.local.40206<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span><span class="m">192</span>.168.1.1.domain:<span class="w"> </span><span class="m">9836</span>+<span class="w"> </span>A?<span class="w"> </span>www.gamedesire.com.<span class="w"> </span><span class="o">(</span><span class="m">36</span><span class="o">)</span>
<span class="m">10</span>:54:17.047721<span class="w"> </span>IP<span class="w"> </span>pma.local.55428<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span><span class="m">192</span>.168.1.1.domain:<span class="w"> </span><span class="m">59517</span>+<span class="w"> </span>PTR?<span class="w"> </span><span class="m">1</span>.1.168.192.in-addr.arpa.<span class="w"> </span><span class="o">(</span><span class="m">42</span><span class="o">)</span>
<span class="m">10</span>:54:17.079337<span class="w"> </span>IP<span class="w"> </span><span class="m">192</span>.168.1.1.domain<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>pma.local.55428:<span class="w"> </span><span class="m">59517</span><span class="w"> </span>NXDomain*<span class="w"> </span><span class="m">0</span>/1/0<span class="w"> </span><span class="o">(</span><span class="m">109</span><span class="o">)</span>
<span class="m">10</span>:54:17.080473<span class="w"> </span>IP<span class="w"> </span><span class="m">192</span>.168.1.1.domain<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>pma.local.40206:<span class="w"> </span><span class="m">9836</span><span class="w"> </span><span class="m">2</span>/2/2<span class="w"> </span>A<span class="w"> </span><span class="m">174</span>.123.95.101,<span class="w"> </span>A<span class="w"> </span><span class="m">174</span>.123.95.100<span class="w"> </span><span class="o">(</span><span class="m">146</span><span class="o">)</span>
^C
<span class="m">4</span><span class="w"> </span>packets<span class="w"> </span>captured
<span class="m">4</span><span class="w"> </span>packets<span class="w"> </span>received<span class="w"> </span>by<span class="w"> </span>filter
<span class="m">0</span><span class="w"> </span>packets<span class="w"> </span>dropped<span class="w"> </span>by<span class="w"> </span>kernel
</pre></div>
<p>From the tcpdump(8) manpage: Name server requests are formatted as:</p>
<div class="highlight"><pre><span></span>src<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>gt<span class="p">;</span><span class="w"> </span>dst:<span class="w"> </span>id<span class="w"> </span>op?<span class="w"> </span>flags<span class="w"> </span>qtype<span class="w"> </span>qclass<span class="w"> </span>name<span class="w"> </span><span class="o">(</span>len<span class="o">)</span>
</pre></div>
<p>I rather suggest reading this manpages yourself – just look for “UDP Name Server Requests” section.</p>Wacom Bamboo Fun in Fedora 152012-01-11T00:00:00+01:002012-01-11T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2012-01-11:2012/Jan/11/wacom-bamboo-fun-in-fedora-15/<p>Today I connected my wife’s Wacom Bamboo Fun CTH-461 to my desktop with Fedora 15 on the 2.6.41.4-1.fc15.x86_64 kernel. It worked like a charm – just plug&play – that’s because I already got installed X drivers for the Wacom:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>rpm<span class="w"> </span>-qa<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-i<span class="w"> </span>wacom<span class="w"> </span>xorg-x11-drv-wacom-0.11.1-3.fc15.x86_64
</pre></div>
<p>If You have issues with Wacom under Fedora 15 – just try to install above drivers and reboot X.</p>
<p>But I really hate this multi – touch when I’d like to draw something with the pen, so I wanted to turn it off. Also I wanted the pen to work only on the first monitor – not on the both (I use dual monitors with extended desktops). We can set all those things with <strong>xwacomset</strong> command. First We have to know our Wacom devices:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>xsetwacom<span class="w"> </span>list<span class="w"> </span>dev<span class="w"> </span>Wacom<span class="w"> </span>Bamboo<span class="w"> </span>Craft<span class="w"> </span>Finger<span class="w"> </span>touch<span class="w"> </span>id:<span class="w"> </span><span class="m">11</span><span class="w"> </span>type:<span class="w"> </span>TOUCH<span class="w"> </span>Wacom<span class="w"> </span>Bamboo<span class="w"> </span>Craft<span class="w"> </span>Finger<span class="w"> </span>pad<span class="w"> </span>id:<span class="w"> </span><span class="m">12</span><span class="w"> </span>type:<span class="w"> </span>PAD<span class="w"> </span>Wacom<span class="w"> </span>Bamboo<span class="w"> </span>Craft<span class="w"> </span>Pen<span class="w"> </span>stylus<span class="w"> </span>id:<span class="w"> </span><span class="m">13</span><span class="w"> </span>type:<span class="w"> </span>STYLUS<span class="w"> </span>Wacom<span class="w"> </span>Bamboo<span class="w"> </span>Craft<span class="w"> </span>Pen<span class="w"> </span>eraser<span class="w"> </span>id:<span class="w"> </span><span class="m">14</span><span class="w"> </span>type:<span class="w"> </span>ERASER
</pre></div>
<p>Now when We have those devices’ names We can turn off the multi touch:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>xsetwacom<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="s1">'Wacom Bamboo Craft Finger touch'</span><span class="w"> </span>touch<span class="w"> </span>off
</pre></div>
<p>When done with multi touch We can set the working space of our tablet to the first display unit. Firsly we have to know current mapping settings:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>docent@docent-desktop<span class="w"> </span>~<span class="o">]</span>$<span class="w"> </span>xsetwacom<span class="w"> </span>get<span class="w"> </span><span class="s2">"Wacom Bamboo Craft Pen stylus"</span><span class="w"> </span>Area<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">14720</span><span class="w"> </span><span class="m">9200</span>
</pre></div>
<p>Now If You want to map the pen to the left monitor, just double the third number:</p>
<div class="highlight"><pre><span></span>xsetwacom<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="s2">"Wacom Bamboo Craft Pen stylus"</span><span class="w"> </span>Area<span class="w"> </span><span class="s2">"0 0 29440 9200"</span>
</pre></div>
<p>Now I had all done and were ready to go with my work :)</p>Postfix: Backup MX - howto2012-01-08T00:00:00+01:002012-01-08T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2012-01-08:2012/Jan/08/postfix-backup-mx-howto/<p><center><img alt="img" src="https://maciej.lasyk.info/images/tux-mail-1ty.gif" /></center></p>
<p>In mail-servers architecture We should always have some backup MX defined for every mail server. It’s very simple why – to have a redundant mail-server architecture and just to be sure, that no emails are returned with an error while our mail-server is having issues.</p>
<p>In the simplest scenario let’s assume that We have only one mail server (mail.somedomain.com). We’d like to start a backup MX server for this. We can do this in a few simple steps:</p>
<h3>Step 1: Backup postfix configuration</h3>
<p>On the backup server We should change some postfix configuration in main.cf file. We should add / change relay_domains, set maximal_queue_lifetime, smtpd_recipient_restrictions and We should create relay_recipient_maps:</p>
<div class="highlight"><pre><span></span><span class="nv">relay_recipient_maps</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>hash:/etc/postfix/relay_recipients
<span class="nv">maximal_queue_lifetime</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>30d
<span class="nv">smtpd_recipient_restrictions</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="o">[</span>...<span class="o">]</span>
<span class="w"> </span>permit_mx_backup
<span class="nv">relay_domains</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$mydestination</span><span class="w"> </span>somedomain.com
<span class="nv">permit_mx_backup_networks</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">128</span>.128.128.0/24<span class="w"> </span><span class="m">201</span>.201.201.0/24
</pre></div>
<p>Now let’s explain the following configuration:</p>
<ul>
<li><strong>relay_recipient_maps</strong> = hash:/etc/postfix/relay_recipients – this is optional, but I advise to use this parameter. It defines a hash table containing valid recipients. If the backup system wouldn’t know all the valid mailboxes it would have to accept all the emails – including spam for non-existing adresses. With knowledge of legal addresses backup server is able to bounce back emails that have invalid recipient set. This does not apply in environment using catchall mailboxes to catch all the emails. I attached a sample relay_recipients file below. Remember to use postmap command after every change in this file: postmap <strong>/etc/postfix/relay_recipients</strong></li>
<li><strong>maximal_queue_lifetime = 30d</strong> – default value for Postfix is 5 days. This number sets the time period in which backup server will try to deliver emails to the main server – so this is maximum time of downtime for main server until mails are bounced back to their original senders with an error.</li>
<li><strong>relay_domains = $mydestination somedomain.com</strong> - this parameter will allow postfix to relay emails for <strong>somedomain.com</strong></li>
<li><strong>permit_mx_backup</strong> - security, see http://www.postfix.org/postconf.5.html#permit_mx_backup</li>
<li><strong>permit_mx_backup_networks</strong> - security, see http://www.postfix.org/postconf.5.html#permit_mx_backup_networks</li>
</ul>
<p>And the sample relay_recipients_file:</p>
<div class="highlight"><pre><span></span>user1@somedomain.com<span class="w"> </span>any_value
user2@somedomain.com<span class="w"> </span>any_value
user3@somedomain.com<span class="w"> </span>any_value
user4@somedomain.com<span class="w"> </span>any_value
user5@somedomain.com<span class="w"> </span>any_value
</pre></div>
<p>So as You see – You should have replicated users addresses on the MX server in the relay_recipients file.</p>
<h3>Step 2: DNS configuration</h3>
<p>Having only one mail server it is enough to have only one MX record in our DNS zone file:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>user@server<span class="w"> </span>~<span class="o">]</span><span class="c1"># dig mx somedomain.com</span>
<span class="p">;;</span><span class="w"> </span>ANSWER<span class="w"> </span>SECTION:
somedomain.com.<span class="w"> </span><span class="m">86400</span><span class="w"> </span>IN<span class="w"> </span>MX<span class="w"> </span><span class="m">10</span><span class="w"> </span>mail.somedomain.com.
</pre></div>
<p>Here We see our only MX record with 10 priority pointing to the A record mail.somedomain.com. In order to create a new record for our backup MX server We should first add a new A record, like:</p>
<div class="highlight"><pre><span></span>mail2.somedomain.com.<span class="w"> </span><span class="m">86400</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">129</span>.129.129.129
</pre></div>
<p>And then We can create a new MX record with lower priority:</p>
<div class="highlight"><pre><span></span>mail.somedomain.com.<span class="w"> </span><span class="m">86400</span><span class="w"> </span>IN<span class="w"> </span>MX<span class="w"> </span><span class="m">20</span><span class="w"> </span>mail2.somedomain.com.
</pre></div>
<h3>Step 3: Flushing messages</h3>
<p>When main MX server is down, and backup server gets some messages to hold those until main server is back – It moves those messages immediately to the flush queue. Now those messages can be delivered via flush daemon, which is run every some time (set in <strong>/etc/postfix/master.cf</strong>):</p>
<div class="highlight"><pre><span></span>flush<span class="w"> </span>unix<span class="w"> </span>n<span class="w"> </span>-<span class="w"> </span>n<span class="w"> </span><span class="m">1000</span>?<span class="w"> </span><span class="m">0</span><span class="w"> </span>flush
</pre></div>
<p>Here the "1000?" stands for 1000 seconds every which flush daemon is activated (until it is not already running – this is why we use here question mark after 1000).</p>
<p>Now we can set how often messages should be flushed via the running flush daemon using the <strong>fast_flush_refresh_time</strong> param (default set to 12h). So every 12h messages that haven’t had redelivery requested are being kicked automaticly.</p>
<p>When our master server is back We could just flush all the messages manually:</p>
<div class="highlight"><pre><span></span>postqueue<span class="w"> </span>-f
</pre></div>
<p>But above command will flush all the messages in the flush queue – this might not be the best solution as the backup MX can be a slave for a bunch of main MX servers – are you sure You would like to flush all those messages from all those servers when only one is back online?</p>
<p>Better solution is to use:</p>
<div class="highlight"><pre><span></span>postqueue<span class="w"> </span>-s<span class="w"> </span>somedomain.com
</pre></div>
<p>Above command will flush only the messages from the given domain – and that’s what We would like to do. But We have to know, that We can use this command only when We have this domain configured as “fast_flush_domains”. Again – We’re lucky, because default <strong>fast_flush_domains</strong> value is:</p>
<div class="highlight"><pre><span></span><span class="nv">fast_flush_domains</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$relay_domains</span>
</pre></div>
<p>And If we configured our somedomain.com as "$relay_domain" - then our flush command will work :) If not then we only have to set:</p>
<div class="highlight"><pre><span></span><span class="nv">fast_flush_domains</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$relay_domains</span><span class="w"> </span>somedomain.com
</pre></div>
<p>And when our main MX comes back again – we can flush this domain on the backup MX – it’s good to be wrapped with some script :)</p>
<p>And we're good to go – from now (after correct DNS entries’ propagation, so in max 72 hours) our backup MX should work and receive emails when master mail server is offline.</p>MySQL statement based replication with triggers, events, procedures, functions and variables - howto2012-01-06T00:00:00+01:002012-01-06T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2012-01-06:2012/Jan/06/mysql-statement-based-replication-with-triggers-events-procedures-functions-and-variables-howto/<p>Before starting statement based replication in MySQL database We have to be aware of some specific behaviors for this environment. This kind of replication (statement based) writes each query that modifies data to the <strong>Binary Log</strong> in order to replicate them on the slave or to use as a <strong>point-in-time recovery</strong> (PITR). Because of this kind of query logging We should be aware how MySQL replication engine behaves with some special queries like triggers, functions, procedures or events.</p>
<h3>Functions</h3>
<p>Function calls are logged directly to Binary Log, so If You forget to create on slave any function that is created on master – You will break your replication and probably You’ll see error like below:</p>
<div class="highlight"><pre><span></span>Last_Error:<span class="w"> </span>Error<span class="w"> </span><span class="s1">'FUNCTION postfix.recount_quota not exist'</span><span class="w"> </span>on<span class="w"> </span>query.<span class="w"> </span>Default<span class="w"> </span>database:<span class="w"> </span><span class="s1">'postfix'</span>.<span class="w"> </span>Query:<span class="w"> </span><span class="s1">'UPDATE user_imap SET quota=(recount_quota())'</span>
</pre></div>
<p>When promoting slave to master no additional steps according to functions are required – everything that is needed is having functions defined in both: master and slave.</p>
<h3>Procedures</h3>
<p>Procedure calls are not replicated as in functions – this is important to know. Only the queries inside the procedures get logged to the Binary Log, so You don’t have to create procedures on slaves.</p>
<p>In order to promote slave to master you should have procedures created on slave – so it is wise to have all the procedures created on both – master and slaves.</p>
<h3>Events</h3>
<p>Events created on master server get replicated to the slave with the <strong>DISABLE ON SLAVE</strong> option – that’s why those events are not reexecuted on every slave in our MySQL architecture and we have no duplicated and corrupted data. MySQL logs only queries from inside the event so only those queries are replicated via Binary Log.</p>
<p>In order to promote slave to master according to events we have to do some more job. I’ve created a simple event below (we create it on the master):</p>
<div class="highlight"><pre><span></span>mysql><span class="w"> </span>CREATE<span class="w"> </span>EVENT<span class="w"> </span>mysql_heartbeat
ON<span class="w"> </span>SCHEDULE<span class="w"> </span>AT<span class="w"> </span>CURRENT_TIMESTAMP<span class="w"> </span>+<span class="w"> </span>INTERVAL<span class="w"> </span><span class="m">1</span><span class="w"> </span>MINUTE
DO<span class="w"> </span>INSERT<span class="w"> </span>UPDATE<span class="w"> </span><span class="sb">`</span>mysql_stat<span class="sb">`</span>.<span class="sb">`</span>heartbeat<span class="sb">`</span><span class="w"> </span>SET<span class="w"> </span><span class="sb">`</span>last<span class="sb">`</span><span class="o">=</span>CURTIME<span class="o">()</span><span class="p">;</span>
</pre></div>
<p>Now it’s replicated on slave via Binary Log – below I’ve placed replication entry for that event from Binary Log:</p>
<div class="highlight"><pre><span></span>CREATE<span class="w"> </span><span class="nv">DEFINER</span><span class="o">=</span><span class="sb">`</span>user<span class="sb">`</span>@<span class="sb">`</span>localhost<span class="sb">`</span><span class="w"> </span>EVENT<span class="w"> </span><span class="sb">`</span>mysql_heartbeat<span class="sb">`</span><span class="w"> </span>ON<span class="w"> </span>SCHEDULE<span class="w"> </span>AT<span class="w"> </span>CURRENT_TIMESTAMP<span class="w"> </span>+<span class="w"> </span>INTERVAL<span class="w"> </span><span class="m">1</span><span class="w"> </span>MINUTE<span class="w"> </span>DO<span class="w"> </span>INSERT<span class="w"> </span>UPDATE<span class="w"> </span><span class="sb">`</span>mysql_stat<span class="sb">`</span>.<span class="sb">`</span>heartbeat<span class="sb">`</span><span class="w"> </span>SET<span class="w"> </span><span class="sb">`</span>last<span class="sb">`</span><span class="o">=</span>CURTIME<span class="o">()</span><span class="p">;</span>
</pre></div>
<p>And how it looks like after replication on the slave:</p>
<div class="highlight"><pre><span></span>CREATE<span class="w"> </span><span class="nv">DEFINER</span><span class="o">=</span><span class="sb">`</span>user<span class="sb">`</span>@<span class="sb">`</span>localhost<span class="sb">`</span><span class="w"> </span>EVENT<span class="w"> </span><span class="sb">`</span>mysql_heartbeat<span class="sb">`</span><span class="w"> </span>ON<span class="w"> </span>SCHEDULE<span class="w"> </span>AT<span class="w"> </span><span class="s1">'2012-01-06 21:12:56'</span><span class="w"> </span>ON<span class="w"> </span>COMPLETION<span class="w"> </span>NOT<span class="w"> </span>PRESERVE<span class="w"> </span>DISABLE<span class="w"> </span>ON<span class="w"> </span>SLAVE<span class="w"> </span>DO<span class="w"> </span>INSERT<span class="w"> </span>UPDATE<span class="w"> </span><span class="sb">`</span>mysql_stat<span class="sb">`</span>.<span class="sb">`</span>heartbeat<span class="sb">`</span><span class="w"> </span>SET<span class="w"> </span><span class="sb">`</span>last<span class="sb">`</span><span class="o">=</span>CURTIME<span class="o">()</span><span class="p">;</span>
</pre></div>
<p>So now with this knowledge a little procedure to promote slave to master using events:</p>
<ul>
<li>Disabling event manager on slave with <strong>SET GLOBAL event_scheduler = OFF;</strong></li>
<li>Enabling all the events with <strong>ALTER EVENT event_name ENABLE</strong> - we have to do this for each event, so writing a little script is very helpful here.</li>
<li>Enabling event manager with <strong>SET GLOBAL event_scheduler = ON;</strong></li>
</ul>
<p>In order to demote back the master to slave You should follow the previous procedure with a little change on ALTER EVENT – here You just need to DISABLE all the events (not ENABLE).</p>
<h3>Triggers</h3>
<p>In order to have triggers running properly on master and slaves You have to define them in both – master and slave servers. MySQL statement based replication replicates only the original query to the Binary Log – not the subsequent triggered statements.</p>
<p>When promoting slave to master no additional steps according to triggers are required – everything is needed is having triggers defined in both: master and slave.</p>
<h3>Mixed triggers / procedures / functions calls</h3>
<p>Let’s imagine that We have a trigger, that triggers a procedure which uses a function call. How will this behave in statement based replication?</p>
<ol>
<li>We should have trigger defined on both: master and slave</li>
<li>We don’t have to have procedure defined on the slave – only on master is enough</li>
<li>We should have function defined on both: master and slave</li>
</ol>
<p>Despite of all – my advice is to keep function, triggers, procedures and events defined on all the servers (masters and slaves) – just to be sure, that We can always promote slave to master without any issues.</p>
<p>And one more thing before finishing this post. If You plan to start replication with just copying FRM, MYI, MYD and InnoDB files You should also dump any functions / triggers and stored procedures on master (or slave) and then import those on the new slave. You can do it (for every database) with:</p>
<div class="highlight"><pre><span></span>mysqldump<span class="w"> </span>--routines<span class="w"> </span>--no-create-info<span class="w"> </span>--no-data<span class="w"> </span>--no-create-db<span class="w"> </span>--skip-opt<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;</span>database<span class="p">&</span>gt<span class="p">;</span><span class="w"> </span><span class="p">&</span>gt<span class="p">;</span><span class="w"> </span>dumpfile.sql
</pre></div>
<p>And recreate those on the new box:</p>
<div class="highlight"><pre><span></span>mysql<span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;</span>database<span class="p">&</span>gt<span class="p">;</span><span class="w"> </span><span class="p">&</span>amp<span class="p">;</span>lt<span class="p">;</span><span class="w"> </span>dumpfile.sql
</pre></div>MySQL tunneling via SSH and error “channel: open failed: connect failed: Connection refused”2011-12-28T00:00:00+01:002011-12-28T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2011-12-28:2011/Dec/28/mysql-tunneling-via-ssh-and-error-channel-open-failed-connect-failed-connection-refused/<p>Lately I wrote a short article about MySQL tunneling via SSH in order to start safe MySQL replication. Afterwards I noticed some problems with creating a new SSH tunnel for MySQL connection on a quite different environment. After creating SSH tunnel and trying to connect via this tunnel to the SSH server I received SSH error on tunnel error-log:</p>
<div class="highlight"><pre><span></span>channel<span class="w"> </span><span class="m">2</span>:<span class="w"> </span>open<span class="w"> </span>failed:<span class="w"> </span>connect<span class="w"> </span>failed:<span class="w"> </span>Connection<span class="w"> </span>refused
</pre></div>
<p>or</p>
<div class="highlight"><pre><span></span>channel<span class="w"> </span><span class="m">3</span>:<span class="w"> </span>open<span class="w"> </span>failed:<span class="w"> </span>connect<span class="w"> </span>failed:<span class="w"> </span>Connection<span class="w"> </span>refused
</pre></div>
<p>and below:</p>
<div class="highlight"><pre><span></span>ERROR<span class="w"> </span><span class="m">2013</span><span class="w"> </span><span class="o">(</span>HY000<span class="o">)</span>:<span class="w"> </span>Lost<span class="w"> </span>connection<span class="w"> </span>to<span class="w"> </span>MySQL<span class="w"> </span>server<span class="w"> </span>during<span class="w"> </span>query
</pre></div>
<p>in the MySQL terminal.</p>
<p>First of all We have to make sure, that our tunnel is working properly, so We just kill the current tunnel and create new one without “-f” and “-N” options:</p>
<div class="highlight"><pre><span></span>ssh<span class="w"> </span>-p<span class="w"> </span><span class="m">2345</span><span class="w"> </span>mysql_tunnel@mysqlmaster-server.com<span class="w"> </span>-L<span class="w"> </span><span class="m">4406</span>:mysqlmaster-server.com:3306
</pre></div>
<p>If everything is ok, then We can assume that tunnel is working fine. We can also try to create another tunnel to some other service on different target port and then just try if this other service is working via the tunnel – just to exclude any problems with SSH tunneling.</p>
<p>My problem was that MySQL was configured in the way it was blocking any connections outside localhost. It is default MySQL configuration – We can achieve it via <strong>my.cnf</strong> entries:</p>
<div class="highlight"><pre><span></span>bind-address<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">127</span>.0.0.1
</pre></div>
<p>or:</p>
<div class="highlight"><pre><span></span>skip-networking
</pre></div>
<p>So in order to make our MySQL accessible via our tunnel We have to comment out the <strong>skip-networking</strong> line and make sure that We are connecting to the correct IP addr in our tunnel. For example If we have in our <strong>my.cnf</strong> this line:</p>
<div class="highlight"><pre><span></span>ssh<span class="w"> </span>-p<span class="w"> </span><span class="m">2345</span><span class="w"> </span>-f<span class="w"> </span>mysql_tunnel@mysqlmaster-server.com<span class="w"> </span>-L<span class="w"> </span><span class="m">4406</span>:127.0.0.1:3306<span class="w"> </span>-N
</pre></div>
<p>(notice that 127.0.0.1 in the above command).</p>
<p>If We would bind our MySQL to some other IP, like:</p>
<div class="highlight"><pre><span></span>bind-address<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">192</span>.168.0.12
</pre></div>
<p>Then We should change our tunneling parameters:</p>
<div class="highlight"><pre><span></span>ssh<span class="w"> </span>-p<span class="w"> </span><span class="m">2345</span><span class="w"> </span>-f<span class="w"> </span>mysql_tunnel@mysqlmaster-server.com<span class="w"> </span>-L<span class="w"> </span><span class="m">4406</span>:192.168.0.12:3306<span class="w"> </span>-N
</pre></div>
<p>After commenting out that <strong>skip-networking</strong> our security depends on IP address we are binding the MySQL to. If it’s local IP addres in DMZ, than there is no security breaches here. Unwise would be to bind to the WAN address and leave MySQL port opened without any SSL encryption or without filtering traffic by the client IP addr…</p>MySQL replication over SSH tunnel2011-12-24T00:00:00+01:002011-12-24T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2011-12-24:2011/Dec/24/mysql-replication-over-ssh-tunnel/<p>Sometimes it is a good decision to replicate between datacenters. It is not for a backup purposes – as replication cannot be used for backups (maybe under some circumstances, but let’s say that for now We’re not thinking about replication as backup solution) – for now We’re using it to just have up2date data in some other datacenter.</p>
<p>General idea to set up this replication is to make a SSH tunnel between those two datacenters and then start transferring data using this secure transport layer. I will call “replication server” – the server that will be slave in our destination and “the master server” will be our master.</p>
<p>I won’t write here how to set up a replication from the scratch. Let’s say that for now there are at least two ways to do it without stopping mysql master (using another slave to take data snapshot or using <a href="http://www.percona.com/doc/percona-xtrabackup/howtos/setting_up_replication.html">Percona Xtrabackup</a>).</p>
<p>Firstly we have to start SSH tunnel. We have to ensure, that this tunnel will keep alive trough any connection problems and will not be killed due to an idle (how come when there is replication stream over this?).</p>
<p>Let’s start with ensuring that our tunnell will keep alive. In SSH client configuration (default: /etc/ssh/ssh_config) We should add the following:</p>
<div class="highlight"><pre><span></span>ServerAliveInterval<span class="w"> </span><span class="m">300</span>
</pre></div>
<p>With above server maintaining the tunnel will send some keep-alive request every 300 seconds to the master (destination) server.</p>
<p>Now We have to open MySQL port on the master (destination) server on WAN interface. This is not secure unless We filter source IP address trying to connect to this port (let’s allow only our slave’s server IP addr. to use this port). For maximum security We can use TCP Wrappers on the master (destination) MySQL server, but this will put some overhead to the server functionality as TCP Wrappers always use some DNS resolution. In my opinion filtering MySQL port based on source IP address is enough.</p>
<p>Now We can start our tunnel:</p>
<div class="highlight"><pre><span></span>ssh<span class="w"> </span>-p<span class="w"> </span><span class="m">2345</span><span class="w"> </span>-f<span class="w"> </span>mysql_tunnel@mysqlmaster-server.com<span class="w"> </span>-L<span class="w"> </span><span class="m">4406</span>:mysqlmaster-server.com:3306<span class="w"> </span>-N
</pre></div>
<p>Let’s explain:</p>
<ul>
<li>-p 2345 – port We are using to connect over SSH (default 22, but should be changed to something else for standard security reasons)</li>
<li>-n – SSH will go to background just before command execution. Make sure, that You have SSH keys exported to the master (destination) server from the slave server / user and You will not have to enter any passwords during creating the tunnel</li>
<li>-L – turns on port forwarding – this is the core of creating SSH tunnel</li>
<li>-N – “do not execute a remote command” – just because We are just forwarding ports :)</li>
</ul>
<p>Now We can test this tunnel. Let’s try to connect to MySQL master from the slave server:</p>
<div class="highlight"><pre><span></span>mysql<span class="w"> </span>-h<span class="w"> </span><span class="m">127</span>.0.0.1<span class="w"> </span>-p<span class="w"> </span><span class="m">4406</span><span class="w"> </span>--user<span class="o">=</span>replication
</pre></div>
<p>We should be able to connect to the master server with above command. And If We really did – then We can use this connection to start the replication.</p>
<p>This is very simple method that should be wrapped with some monitoring, scripts that will create SSH tunnels automatically when the original tunnel dies or after server crash. We should also remember, that replication lags can be quite high using this technique – everything depends on connection quality and number of writes on master that will have to be replicated on slave. In order to tune this method of replication It can be good to use statement-based replication – because in many cases this method use a bunch less number of kilobytes to transmit replication data to the slave.</p>Calculating file size in bash2011-12-09T00:00:00+01:002011-12-09T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2011-12-09:2011/Dec/09/calculating-file-size-in-bash/<p>When we face the problem of calculating size of some files then we often think of du.. But how could we count size of files found by find? We could do this with this simple one-liner:</p>
<div class="highlight"><pre><span></span>find<span class="w"> </span>/somewhere<span class="w"> </span>-name<span class="w"> </span><span class="s1">'some_files*'</span><span class="w"> </span>-exec<span class="w"> </span>wc<span class="w"> </span>-c<span class="w"> </span><span class="s1">'{}'</span><span class="w"> </span>+
</pre></div>
<p>Find command is simple here. We use here also a POSIX solution – ending “exec” with a plus-sign not a semicolon. This form groups the finding results into sets and then run commands on whole that set. And that’s why we use it here – our <strong>wc -c</strong> command is run on a whole set of results giving us in the end line the total size of all the files:</p>
<div class="highlight"><pre><span></span>root@docent<span class="w"> </span>log<span class="o">]</span><span class="c1"># find . -name 'maillog*' -exec wc -c '{}' +<br /></span>
<span class="w"> </span><span class="m">6847</span><span class="w"> </span>./maillog-20111120<br<span class="w"> </span>/>
<span class="w"> </span><span class="m">6137</span><span class="w"> </span>./maillog<br<span class="w"> </span>/>
<span class="w"> </span><span class="m">6580</span><span class="w"> </span>./maillog-20111204<br<span class="w"> </span>/>
<span class="w"> </span><span class="m">5028</span><span class="w"> </span>./maillog-20111113<br<span class="w"> </span>/>
<span class="w"> </span><span class="m">6424</span><span class="w"> </span>./maillog-20111127<br<span class="w"> </span>/>
<span class="m">31016</span><span class="w"> </span>total
</pre></div>
<p>We could shorten that a little:</p>
<div class="highlight"><pre><span></span>root@docent<span class="w"> </span>log<span class="o">]</span><span class="c1"># find . -name 'maillog*' -exec wc -c '{}' + | tail -1<br /></span>
<span class="m">31016</span><span class="w"> </span>total
</pre></div>
<p>And If we would like to have only the number:</p>
<div class="highlight"><pre><span></span>root@docent<span class="w"> </span>log<span class="o">]</span><span class="c1"># find . -name 'maillog*' -exec wc -c '{}' + | tail -1 | cut -d' ' -f 1<br /></span>
<span class="m">31016</span>
</pre></div>Grep is your friend2011-12-07T00:00:00+01:002011-12-07T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2011-12-07:2011/Dec/07/grep-is-your-friend/<p><strong>GREP</strong> stands for Global Regular Expression Print. I think that every sysop loves <strong>grep</strong>, <strong>grepping</strong> and anything that has something in common with <strong>grep</strong> – this tool makes our lives really easier ;) If You’re not convinced than I think You’re in a good place – maybe the following text will convince You :)</p>
<h3>Excluding irrelevant words</h3>
<p>Sometimes We have to grep for some word but We have to exclude some irrelevant string. E.g. let’s grep for ‘index.html’ but let’s also exclude ‘404’ from this:</p>
<div class="highlight"><pre><span></span>grep<span class="w"> </span><span class="s1">'index.html'</span><span class="w"> </span>access.log<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-v<span class="w"> </span><span class="m">404</span>
</pre></div>
<h3>egrep</h3>
<p>Eextended grep, same as <strong>grep -e</strong> or <strong>grep –regexp=</strong>, allows us to do more powerful search including regular expressions with metacharacters like +, ?, | and ():</p>
<div class="highlight"><pre><span></span>egrep<span class="w"> </span><span class="s2">"html|cgi"</span><span class="w"> </span>access.log
</pre></div>
<h3>Counting results</h3>
<p>If we just want to know the number of lines that matched our query – We would use:</p>
<div class="highlight"><pre><span></span>grep<span class="w"> </span>-c<span class="w"> </span><span class="s1">'index.html'</span><span class="w"> </span>access.log
</pre></div>
<h3>Case Insensitive search</h3>
<p>By default grep is case sensitive, If we want to make case insensitive search than we use:</p>
<div class="highlight"><pre><span></span>grep<span class="w"> </span>-i<span class="w"> </span><span class="s1">'Index'</span><span class="w"> </span>access.log
</pre></div>
<h3>Matching eXact word only</h3>
<p>By default grepping for <strong>Word</strong> will return lines containing <strong>SomeWord</strong> and <strong>WordSomething</strong> (doesn't take care of word boundaries). If we would like to find only those lines containing exact word <strong>Word</strong> We should use:</p>
<div class="highlight"><pre><span></span>grep<span class="w"> </span>-x<span class="w"> </span><span class="s1">'404'</span><span class="w"> </span>access.log
</pre></div>
<p><strong>grep -w</strong> could be also useful above.</p>
<h3>Matching left and right side of the word</h3>
<p>To search for instances of string matching Word in the end or start (boundaries) We use <strong>\<</strong> or <strong>></strong></p>
<p>Below would match any word starting with access, like access_entry:</p>
<div class="highlight"><pre><span></span>grep<span class="w"> </span><span class="s1">'\<access'</span>
</pre></div>
<p>Below would match any word ending with error, like general_error:</p>
<div class="highlight"><pre><span></span>grep<span class="w"> </span><span class="s1">'error\>'</span>
</pre></div>
<h3>Showing context results</h3>
<p>Sometimes We would like to grep for some errors in logs, but we also would like to view the context of that log entry – e.g. grepping for ‘Relay access denied‘ in Postfix logs to see If that error is occurring with some pattern:</p>
<div class="highlight"><pre><span></span>grep<span class="w"> </span>--context<span class="o">=</span><span class="m">3</span><span class="w"> </span><span class="s1">'Relay access denied'</span><span class="w"> </span>maillog
</pre></div>
<h3>zgrep - grepping through compressed files</h3>
<p>This one would <strong>grep</strong> in the compressed gzip file – just like <strong>gunzip -c flog.gz | grep Word</strong>:</p>
<div class="highlight"><pre><span></span>zgrep<span class="w"> </span><span class="s1">'Relay access denied'</span><span class="w"> </span>maillog3.gz
</pre></div>
<h3>Coloring matched words</h3>
<p>We can highlight our matched words with some color (check man page to see how to set exact color):</p>
<div class="highlight"><pre><span></span>grep<span class="w"> </span>--color<span class="w"> </span><span class="s1">'Relay access denied'</span><span class="w"> </span>maillog3.gz
</pre></div>Hello there!2011-12-05T00:00:00+01:002011-12-05T00:00:00+01:00Maciej Lasyktag:maciej.lasyk.info,2011-12-05:2011/Dec/05/hello-there/<p>Hello there, I’ve just deployed this blog – so this is my “Hello World” message :) I’m keeping my fingers crossed for my consistently creating new posts here ;)</p>
<p>May the Tux be with me!</p>
<p><center><img alt="old-blog-1" src="https://maciej.lasyk.info/images/old-blog-1.jpg" /></center></p>