Hi all,I am quite new to databricks. Overall I enjoyed the experience so far, but now ran into a problem, I was not able to find an acceptable solution.Here is my setup: I have a bunch of s3 buckets, and need to put the data into databricks, preferab...
I' m trying to print out number of rows in the batch, but seems it doesn't work properly. I have 1 node compute optimized cluster and run in notebook this code:# Logging the row count using a streaming-friendly approach def log_row_count(batch_df, ba...
while mount to the storage account using this below code dbutils.fs.mount( source=f"wasbs://{cointainer_name}@{storage_name}.blob.core.windows.net", mount_point=f"/mnt/{cointainer_name}", extra_configs={f"fs.azure.account.key.{storage_name}.blob.c...
I’m studying for the PL-300 exam and would love some advice on how to optimize Power BI reports and dashboards for better performance. Specifically, I’m interested in:Techniques for improving report load times and responsiveness.Best practices for ma...
Hi @jamson ,Here's a good article that answers your question. I think author did a pretty good job - many of his advice I apply on everyday job.How I Tuned Databricks Query Performance from Power BI Desktop: A Personal Journey. | by Brahmareddy, The ...
Hi, I am supposed to create transformation notebook. But i am having trouble when i am trying to save the transformed file into blob storage. I didn't use any layer, just the layer which i am performing transformation in. if i use wasbs i receive dif...
Hi @yagmur ,Did you assigned required permission to service principal on storage account?And make sure you're configuring connection to storage account in proper way. You should have something similiar to the code below: configs = { "fs.a...
I want to pass some context information to the delta live tables pipeline when calling from Azure Data Factory. I know the body of the API call supports Full Refresh parameter but I wonder if I can add my own custom parameters and how this can be re...
@rubenesanchez . I am also not able to fetch the details passed from ADF to DLT notebook. Were you able to resolve?
Hi,So previously I was using Old Preview Databricks API version. Now I have switched to v1 of the API which usage Execute a SQL statement POST /api/2.0/sql/statements/I wanted to know how to pass the parameter class when my value is list of string fo...
Hi @szymon_dybczak Any idea how to better handle IN operator with a list in new API version
Facing issues while Importing dlt library in Databricks Runtime 14.3 LTS. Previously while using the Runtime 13.1 The `import dlt` was working fine but now when updating the Runtime version to 14.3 LTS it is giving me error.
I ran the code block below and now I can't find the file. Where would this get saved since no location was specified?
/Workspace/Users/your.email@companyname.com/myfile.txt
I have two notebooks created for my Delta Live Table pipeline. The first is a utils notebook with functions I will be reusing for other pipelines. The second contains my actual creation of the delta live tables. I added both notebooks to the pipeline...
Hi Dave,You can solve this by putting your utils into a python file and referencing your .py file in the DLT notebook. I provided a template for the python file below:STEP 1: #import functions from pyspark.sql import SparkSession import IPython dbut...
Hi Databricks community,I'm facing a challenge extracting JSON data from Elasticsearch in Azure Databricks efficiently, maintaining header information.Previously, I had to use RDDs for parallel extraction, but they're no longer supported in Databrick...
Hi,I wanted to ask if anyone is using Databricks SQL Warehouse as backend for small to large scale web application? What are your thoughts about it, specially what Databricks team thinks of it ?Kind Regards,A
Integrating Databricks SQL Warehouse as the backend for a web application involves several steps to handle data processing, querying, and analytics. Here's a simplified guide:Set Up Databricks SQL WarehouseCreate an Account: Sign up for a Databricks ...
Hi,I have a string column containing a number in EU format, has comma instead of dot, e.g. 10,35I need to convert this string into a proper decimal data type as part data transformation into the target table.I could do it as below by replacing the ",...
Hi @Harsha777 ,Your solution looks good!However, you may try also to_number function, but unfortunately still will need to first to replace "," with ".". from pyspark.sql.functions import to_number, regexp_replace, lit data = [("10,6523",), ("10,23"...
I am facing issues while Importing dlt library in Databricks Runtime 14.3. Previously while using the Runtime 13.1 The `import dlt` was working fine but now when updating the Runtime it is giving me error.This is the Cluster's Configuration Also ...
@Kaniz_Fatma Do you have any solution for this above problem ? I saw you reply in this link https://community.databricks.com/t5/data-engineering/no-module-named-dlt/td-p/21105. so i just ask you. Thank you !
I there,I have a scenario where the source XML files may have all the fields or may be 80% of fields in next run. How to we load the files in Delta tables which should handle the XML files with all field lists and also with few fields only. In smalle...
Auto Loader is not acceptable solution in my case. I tried to make an empty table using XSD file and then load the data frame. Some how it worked to meet the objective.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group