- 5776 Views
- 1 replies
- 0 kudos
Spot instances usage in Azure Databricks
Hi all,as per the below article -https://community.databricks.com/t5/technical-blog/optimize-costs-for-your-data-and-ai-workloads-with-azure-and-aws/ba-p/662411. it is possible to choose the number of spot instances using 'availability' parameter. Bu...
- 5776 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @noorbasha534, Thanks for your question! 1. 'availability' Parameter: The 'availability' parameter in Azure Databricks controls whether the compute uses on-demand or spot instances. The values for this parameter are: ON_DEMAND_AZURE: This value...
- 0 kudos
- 4891 Views
- 11 replies
- 3 kudos
Accessing delta tables using API outside azure (Workiva)
I need to access delta tables with API outside azure using in a reporting tool workiva with using the connector. Can someone able to provide the details on how I can achieve it
- 4891 Views
- 11 replies
- 3 kudos
- 3 kudos
I need to retrieve delta tables using an API outside of Azure without requiring a connection using a reporting tool named Workiva. I would appreciate it if someone could tell me exactly how to do that.
- 3 kudos
- 42863 Views
- 4 replies
- 2 kudos
Understanding Used Memory in Databricks Cluster
Hello, I wonder if anyone could give me any insights regarding used memory and how could I change my code to "release" some memory as the code runs. I am using a Databricks Notebook.Basically, what we need to do is perform a query, create a spark sql...
- 42863 Views
- 4 replies
- 2 kudos
- 2 kudos
I have exactly the same kind of problem.I really do not understand why my driver goes out of memory meanwhile I do not cache anything in Spark.Since I don't cache anything, I expect references to objects that are not used anymore to be freed.Even a s...
- 2 kudos
- 4152 Views
- 1 replies
- 2 kudos
Brickflow - An opinionated python framework to help build and deploy Databricks workflows at scale
Hello, Databricks Community, I’m thrilled to introduce “Brickflow,” our innovative solution designed to streamline the development and deployment of workflows at scale, leveraging straightforward Python coding. This tool has already achieved a signif...
- 4152 Views
- 1 replies
- 2 kudos
- 2 kudos
Brickflow is indeed an awesome tool to work w/ Databricks!
- 2 kudos
- 1405 Views
- 2 replies
- 0 kudos
Query related to Z ordering
I have join on two large tables.If i apply Z ordering on 3 columns for both the tables ,I am joining two tables on the basis on the same 3 columns used for Z ordering ,Will i get any benefit of Z ordering on performance when i use Joins here .So as p...
- 1405 Views
- 2 replies
- 0 kudos
- 0 kudos
I have asked this query because w.r.t Predicate query push down to storage .So here as part of joins . Data needs to be loaded into memory first and then join is performed . So How does Z order help here if it can't skip the data being fetched from s...
- 0 kudos
- 5323 Views
- 1 replies
- 0 kudos
Resolved! getting short of error while mount to storage account.
while mount to the storage account using this below code dbutils.fs.mount( source=f"wasbs://{cointainer_name}@{storage_name}.blob.core.windows.net", mount_point=f"/mnt/{cointainer_name}", extra_configs={f"fs.azure.account.key.{storage_name}.blob.c...
- 5323 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @kumarPatra_07 , Greetings! From the above code which you have shared, I could see that you are using a WASBS driver to mount the storage and as of now, WASB is already deprecated. Reference document : https://learn.microsoft.com/en-us/azure/dat...
- 0 kudos
- 4533 Views
- 1 replies
- 0 kudos
Log data from reports in PowerBI
Where to find log data from PowerBI? I need to find what tables are being used in my PowerBI reports that are pointing to Databricks. I tried system.access.audit but I'm not finding new data when I refresh my report
- 4533 Views
- 1 replies
- 0 kudos
- 0 kudos
@ksenija To enable ODBC logging in PowerBI, go to C:\Program Files\Microsoft Power BI Desktop\bin\ODBC Drivers\Simba Spark ODBC Driver folder and create or edit the file microsoft.sparkodbc.ini and update as below: [Driver] LogLevel=6 LogPath=<...
- 0 kudos
- 5478 Views
- 4 replies
- 2 kudos
Resolved! Timezone in silver tables
Hello,What is the best practice in Databricks for storing DateTime data in silver layer tables, considering the source data is in AEST and we store it in UTC by default?Thanks
- 5478 Views
- 4 replies
- 2 kudos
- 2 kudos
@Freshman wrote:Hello,What is the best practice in Databricks for storing DateTime data in silver layer tables, considering the source data is in AEST and we store it in UTC by default?ThanksThe best practice for storing DateTime data in the Silver l...
- 2 kudos
- 2282 Views
- 1 replies
- 1 kudos
Resolved! if-else statement in DAB YAML file
Hi.Is it possible to use a "better" way to override the "git_branch" key's value on the right file (which is the resource-yaml-file)?Or a different way, like an "if-else" statement. I'd like to have it all in the resource-yaml-file instead of overrid...
- 2282 Views
- 1 replies
- 1 kudos
- 1 kudos
I think your approach is correct using targets and ${var.git_branch} variables. At the moment there is no if-else statement, but will investigate internally if there is a better way.
- 1 kudos
- 1269 Views
- 2 replies
- 0 kudos
Read task level parameters in python
I am creating Databricks jobs and tasks using a python package. I have defined a task level parameter and would like to reference it in my script using sys.argv. How can I do that ?
- 1269 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks, but the link works for notebook. I have a python package run as python wheel and am wondering how to access the parameters. When I run the job its not able to understand the task level parameters in sys.argv
- 0 kudos
- 2208 Views
- 2 replies
- 4 kudos
DLT multiple source table to single silver table generating unexpected result
Hi,I´ve been trying this all day long. I'm build a POC of a pipeline that would be used on my everyday ETL.I have two initial tables, vendas and produtos, and they are as the following:vendas_rawvenda_idproduto_iddata_vendaquantidadevalor_totaldth_in...
- 2208 Views
- 2 replies
- 4 kudos
- 4 kudos
When dealing with Change Data Capture (CDC) in Delta Live Tables, it's crucial to handle out-of-order data correctly. You can use the APPLY CHANGES API to manage this. The APPLY CHANGES API ensures that the most recent data is used by specifying a co...
- 4 kudos
- 2455 Views
- 4 replies
- 1 kudos
How to use Widgets with SQL Endpoint in Databricks?
I' trying to use widgets with SQL Endpoints but I'm encountering an error, whereas they work seamlessly with Databricks Interactive Cluster. While query parameters can substitute widgets in SQL endpoints, but I specifically require dropdown and multi...
- 2455 Views
- 4 replies
- 1 kudos
- 1 kudos
Got it, let me test, i think there is no specific way to do it, but if you add the option All as the first on the list it should select it
- 1 kudos
- 2136 Views
- 3 replies
- 1 kudos
Resolved! Issue with filter_by in Databricks SQL Query History API (/api/2.0/sql/history/queries)
Hi everyone,I'm trying to use the filter_by parameter in a GET request to /api/2.0/sql/history/queries, but I keep getting a 400 Bad Request error. When I use max_results, it works fine, but adding filter_by causes the request to fail.Example value f...
- 2136 Views
- 3 replies
- 1 kudos
- 1610 Views
- 1 replies
- 1 kudos
Resolved! Extracting data from GCP BigQuery using Foreign Catalog
On Databricks, I have created a connection type Google Query and tested the connection successfully. I have then created a foreign catalog from the connection to a Google BigQuery project. I can see all the data sets and tables in the Foreign Catalog...
- 1610 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @GinoBarkley, Could you please advise which DBR version are you using in your personal access mode? I see this requirments: Databricks clusters must use Databricks Runtime 16.1 or above and shared or single user access mode. SQL warehouses must ...
- 1 kudos
- 5688 Views
- 6 replies
- 2 kudos
'from_json' spark function not parsing value column from Confluent Kafka topic
For one of badge completion, it was mandatory to complete a Spark Streaming Demo Practice.Due to the absence of a Kafka broker setup required for the demo practice, I configured a Confluent Kafka cluster and made several modifications to the Spark sc...
- 5688 Views
- 6 replies
- 2 kudos
- 2 kudos
I am not sure if I read the full explanation but how about this : df .withColumn('value_str', F.decode(F.col('value'), 'utf-8')) .withColumn('value_json', F.explode(F.from_json(F.col('value_str'), json_schema))) .select('value_json.*'...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 577 | |
| 349 | |
| 287 |
