- 2368 Views
- 6 replies
- 8 kudos
From Associate to Professional: My Learning Plan to ace all Databricks Data Engineer Certifications
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
- 2368 Views
- 6 replies
- 8 kudos
- 8 kudos
This is great! I have worked with Databricks for almost three years and has decided to pursue the Databricks Engineer Professional certification. This will certainly help setting up an effective plan.
- 8 kudos
- 1081 Views
- 1 replies
- 0 kudos
How to convert txt files to delta tables
Hello members of Databricks's comunity,I am currently working on a project where we collect data from machines, that data is in .txt format. The data is currently in an Azure container, I need to clean the files and convert them to delta tables, how ...
- 1081 Views
- 1 replies
- 0 kudos
- 0 kudos
https://docs.databricks.com/en/ingestion/add-data/upload-data.html
- 0 kudos
- 595 Views
- 0 replies
- 0 kudos
RocksDB for storing state stream
Now, you can keep the state of stateful streaming in RocksDB. For example, retrieving keys from memory to check for duplicate records inside the watermark is now faster. #databricks
- 595 Views
- 0 replies
- 0 kudos
- 688 Views
- 0 replies
- 1 kudos
State of stateful streaming
For stateful streaming in #databricks, you can now easily read what is in the state.
- 688 Views
- 0 replies
- 1 kudos
- 1563 Views
- 4 replies
- 0 kudos
Unable to mount GCS bucket with underscores in the name
I have two buckets with the same configurations and labels.One is named my-bucket and the other is my_bucket. I am able to mount my-bucket but get an opaque error message when trying to mount my_bucket. Is this known/expected behavior? Are underscore...
- 1563 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @legobricks , Curious on the error that you are getting. However, for GCS - https://cloud.google.com/storage/docs/buckets#naming I do see underscores are allowed but there is also a note below: You can use a bucket name in a DNS record as part of ...
- 0 kudos
- 2141 Views
- 2 replies
- 1 kudos
Debug stream
One of the easiest ways to debug a stream is to stream it to memory and query it with SQL #databricks
- 2141 Views
- 2 replies
- 1 kudos
- 1153 Views
- 0 replies
- 0 kudos
Financial Crime detection with the help of Apache Spark, Data Mesh and Data Lake
For those interested in Data Mesh and Data Lakes for FinCrime detection:Data mesh is a relatively new architectural concept for data management that emphasizes domain-driven data ownership and self-service data availability. It promotes the decentral...
- 1153 Views
- 0 replies
- 0 kudos
- 1150 Views
- 0 replies
- 0 kudos
Hiring Databricks Data Architect roles
Hi,I am a recruiter and I am looking for places to post some data bricks I have coming out. I have several fully remote, high-level data databricks, architect roles. Of course I will post to LinkedIn, but I was just curious if there are any other pla...
- 1150 Views
- 0 replies
- 0 kudos
- 1806 Views
- 1 replies
- 2 kudos
Databricks Notebook Workflow
Exciting news for Databricks users! The ability to view job details within the notebook workflow section, particularly for multithreaded jobs, is available now. Instead of manually inspecting each job for failures, this feature enables us to swiftly ...
- 1806 Views
- 1 replies
- 2 kudos
- 2 kudos
Thank you for sharing
- 2 kudos
- 2218 Views
- 3 replies
- 5 kudos
🌟 Welcome to the Knowledge Sharing Hub! 🌟
Are you passionate about sharing your discoveries and insights with the world? Look no further! Our Knowledge Sharing Hub is the perfect space for you to showcase your research and connect with like-minded individuals across the globe. Here's why you...
- 2218 Views
- 3 replies
- 5 kudos
- 948 Views
- 1 replies
- 1 kudos
Spot databricks VMs - eviction rates
Before using Spot machines in #databricks, it's a good idea to check their eviction rates in your region. Azure Resource Graph Explorer and that simple query will help. SpotResources | where type =~ 'microsoft.compute/skuspotevictionrate/location' ...
- 948 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for sharing this @Hubert-Dudek
- 1 kudos
- 1412 Views
- 0 replies
- 1 kudos
Installing CrowdStrike Falcon Sensor on Databricks Workers
Greetings,Does anyone here have experience deploying the CrowdStrike Falcon sensor on Databricks worker instances? For context, the cluster is deployed in AWS and we use a Databricks Ubuntu 20.04 AMI. Databricks allows adding a bootstrap/startup scri...
- 1412 Views
- 0 replies
- 1 kudos
- 2060 Views
- 0 replies
- 1 kudos
Post: Lakehouse Federation - Databricks
Lakehouse Federation - Databricks In the world of data, innovation is constant. And the most recent revolution comes with Lakehouse Federation, a fusion between data lakes and data warehouses, taking data manipulation to a new level. This advancement...
- 2060 Views
- 0 replies
- 1 kudos
- 790 Views
- 0 replies
- 0 kudos
Feedback request for Gradient, a tool to help optimize and monitor jobs automatically
Hi Everyone,We built Gradient, a tool to automatically optimize and monitor Databricks jobs to hit your business objectives of cost or runtime.Gradient works by applying a reinforcement ML model to automatically learn and custom tune your jobs cluste...
- 790 Views
- 0 replies
- 0 kudos
- 665 Views
- 0 replies
- 0 kudos
Understand why your jobs' performances are changing over time
Hi Folks -We released a new metrics view for databricks jobs in Gradient, which helps track and plot the metrics below over time to help engineers understand what's going on with their jobs over time.Job cost (DBU + Cloud fees)Job RuntimeNumber of co...
- 665 Views
- 0 replies
- 0 kudos
- 2469 Views
- 1 replies
- 1 kudos
Jonathan Frankel at Sigma talk
Hi @Sujitha Just to follow up on your suggestion to share my takeaways from Jonathan Frankel's talk at Sigma in NYC. The key ideas I came away with is:Building in-house custom models is more than just possible, there's advantages to itThere's danger...
- 2469 Views
- 1 replies
- 1 kudos
- 1 kudos
@Danny_Lee This is super insightful! Really appreciate your time to share your key takeaways with us.
- 1 kudos
