- 6146 Views
- 2 replies
- 6 kudos
Resolved! How to pass a variable which holds a value to child notebook using run command
Hello,We have 3 notebooks as below. And trying to send a variable where we pass the value to it via job to child notebook with %run commandNotebooks:notebook_par,notebook_child1 and notebook_child2.In parent note book we are calling notebook_child1 u...
- 6146 Views
- 2 replies
- 6 kudos
- 6 kudos
Hey there @Sailaja B​ How's it going?Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Chee...
- 6 kudos
- 53510 Views
- 10 replies
- 8 kudos
When to use delta lake versus relational database as a source for BI reporting?
Assume all of your data exists in delta tables and also in SQL server so you have a choice to report from either. Can someone share thoughts on "In what scenario you would not want report created from delta table and instead use the traditional rel...
- 53510 Views
- 10 replies
- 8 kudos
- 8 kudos
Hi @Kaniz Fatma​ - I would like a follow-up on @prasad vaze​ question regarding unsupported referential integrity. How does one work around that, using best practices as Databricks sees it?
- 8 kudos
- 1068 Views
- 0 replies
- 0 kudos
greycupwinner.livejournal.com
Thanks for the information that you share with us, GREY CUP 2022 I really like your content about this.
- 1068 Views
- 0 replies
- 0 kudos
- 1961 Views
- 1 replies
- 0 kudos
In Databricks, when saving a query, the changes do no persist. Any advice?
When returning to the query, the changes have reverted.Is this because the query is open by another user?How can this be resolved? #databricks
- 1961 Views
- 1 replies
- 0 kudos
- 0 kudos
We ARE using the save feature. But we are still loosing changes. Is this because we have multiple Databricks tabs open in separate Chrome browsers? Across 2 users.
- 0 kudos
- 30055 Views
- 9 replies
- 8 kudos
Resolved! Failed to start cluster
Hi, I ran the cluster more than 5-6 times with it failing to start since this past morning (about 11-12 hours now) since i'm facing this problem.Attaching screenshot below and also typing in case someone comes from the web to this thread in future.Pr...
- 30055 Views
- 9 replies
- 8 kudos
- 8 kudos
The cluster will be able to start and the nodes will automatically obtain the updated cluster configuration data.
- 8 kudos
- 3454 Views
- 0 replies
- 0 kudos
Converting a text widget to a list
Hi, Been working on some parallel notebook code, which I have ported to python from the example on the DB website and added some exception handling and that works fine. What I would like to do is paramterise the input but am not succeeding as the fun...
- 3454 Views
- 0 replies
- 0 kudos
- 5153 Views
- 2 replies
- 2 kudos
How to restore DatabricksRoot(FileStore) data after Databricks Workspace is decommissioned?
My Azure Databricks workspace was decommissioned. I forgot to copy files stored in the DatabricksRoot storage (dbfs:/FileStore/...).Can the workspace be recommissioned/restored? Is there any way to get my data back?Also, is there any difference betwe...
- 5153 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @Anmol Deep​ Please submit a support request ASAP, so we can restore the deleted workspace. You can recover artifacts from the workspace.
- 2 kudos
- 5834 Views
- 4 replies
- 1 kudos
Does Databricks have a google cloud Big Query equivalent of --dry_run to estimate costs before executing?
Databricks uses DBU's as a costing unit whether based onto of AWS/Azure/GCP and I want to know if Databricks has a google cloud Big Query equivalent of --dry_run for estimating costs? https://cloud.google.com/bigquery/docs/estimate-costs
- 5834 Views
- 4 replies
- 1 kudos
- 1 kudos
Not that I know of.Google uses number of bytes read to determine the cost.Databricks uses DBU. The number of DBU's spent is not only dependent on the amount of bytes read (the more you read, the longer the program will run probably), but also the typ...
- 1 kudos
- 4122 Views
- 3 replies
- 1 kudos
Loading Dimensions including SCDType2
I have a customer dimension and for every incremental load I am applying type2 or type1 to the dimension.This dimension is based off a silver table in my delta lake where I am applying a merge statement.What happens if I need to go back and track ad...
- 4122 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks werners,I was informed you could essentially recreate a type 2 dimensions from scratch, without reading the files 1 by 1, using the delta lake time shift. However, this doesn't seem to be the case and the only way to create this is to incremen...
- 1 kudos
- 2375 Views
- 0 replies
- 2 kudos
Spark Structured Streaming: How to run N queries on each window
I have timeseries data in k Kafka topics. I would like to read this data into windows of length 10 minutes. For each window, I want to run N SQL queries and materialize result. The specific N queries to run depends on the kafka topic name. How should...
- 2375 Views
- 0 replies
- 2 kudos
- 12691 Views
- 8 replies
- 6 kudos
Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster
We've setup a premium workspace with passthrough credentials cluster , while they do work and access my adls gen 2 storageI can't make it install a library on the cluster from there. and keeping getting"Library installation attempted on the driver no...
- 12691 Views
- 8 replies
- 6 kudos
- 6 kudos
Sorry I can't figure this out, the link you've added is irrelevant for passthrough credentials, if we add it the cluster won't be passthrough, Is there a way to add this just for a specific folder? while keeping passthrough for the rest?
- 6 kudos
- 1570 Views
- 0 replies
- 4 kudos
Databricks Office Hours Register for Office Hours to participate in a live Q&A session with Databricks experts! Our next event is scheduled for Ju...
Databricks Office HoursRegister for Office Hours to participate in a live Q&A session with Databricks experts! Our next event is scheduled for June 22nd from 8:00 am - 9:00am PT.This is your opportunity to connect directly with our experts to ask any...
- 1570 Views
- 0 replies
- 4 kudos
- 6432 Views
- 4 replies
- 2 kudos
Resolved! Unable to delete a DBFS folder
Hello everyone,I've created by error a DBFS folder named : ${env]But when I run this command :dbutils.fs.rm("/mnt/${env]")It returns me this error : java.net.URISyntaxException: Illegal character in path at index 12: /mnt/$%7Benv]How can I do please ...
- 6432 Views
- 4 replies
- 2 kudos
- 5591 Views
- 2 replies
- 3 kudos
Why does my Notebook fails if I try to load a function from another Notebook in Repos in Databricks
My function in func.pydef lower_events(df): return df.withColumn("event",f.lower(f.col("event")))My main notebook import pyspark.sql.functions as f from pyspark.sql.functions import udf, col, lower import sys sys.path.append("..") from folder.func...
- 5591 Views
- 2 replies
- 3 kudos
- 3 kudos
@Kaniz Fatma​ https://community.databricks.com/s/question/0D58Y00008ouo6xSAA/how-to-fetch-environmental-variables-saved-in-one-notebook-into-another-notebook-in-databricks-repos-and-notebooksCan you please look into this
- 3 kudos
- 5132 Views
- 2 replies
- 2 kudos
Resolved! Importing python function with spark.read.jdbc in to Repos
Hi all!Before we used Databricks Repos we used the run magic to run various utility python functions from one notebook inside other notebooks, fex like reading from a jdbc connections. We now plan to switch to repos to utilize the fantastic CI/CD pos...
- 5132 Views
- 2 replies
- 2 kudos
- 2 kudos
Thats...odd. I was sure I had tried that, but now it works somehow. I guess it has to be that now I did it with double quotation marks. Thanks anyway! Works like a charm.
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
