- 4004 Views
- 4 replies
- 1 kudos
Does Databricks have a google cloud Big Query equivalent of --dry_run to estimate costs before executing?
Databricks uses DBU's as a costing unit whether based onto of AWS/Azure/GCP and I want to know if Databricks has a google cloud Big Query equivalent of --dry_run for estimating costs? https://cloud.google.com/bigquery/docs/estimate-costs
- 4004 Views
- 4 replies
- 1 kudos
- 1 kudos
Not that I know of.Google uses number of bytes read to determine the cost.Databricks uses DBU. The number of DBU's spent is not only dependent on the amount of bytes read (the more you read, the longer the program will run probably), but also the typ...
- 1 kudos
- 3182 Views
- 3 replies
- 1 kudos
Loading Dimensions including SCDType2
I have a customer dimension and for every incremental load I am applying type2 or type1 to the dimension.This dimension is based off a silver table in my delta lake where I am applying a merge statement.What happens if I need to go back and track ad...
- 3182 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks werners,I was informed you could essentially recreate a type 2 dimensions from scratch, without reading the files 1 by 1, using the delta lake time shift. However, this doesn't seem to be the case and the only way to create this is to incremen...
- 1 kudos
- 1757 Views
- 0 replies
- 2 kudos
Spark Structured Streaming: How to run N queries on each window
I have timeseries data in k Kafka topics. I would like to read this data into windows of length 10 minutes. For each window, I want to run N SQL queries and materialize result. The specific N queries to run depends on the kafka topic name. How should...
- 1757 Views
- 0 replies
- 2 kudos
- 10608 Views
- 8 replies
- 6 kudos
Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster
We've setup a premium workspace with passthrough credentials cluster , while they do work and access my adls gen 2 storageI can't make it install a library on the cluster from there. and keeping getting"Library installation attempted on the driver no...
- 10608 Views
- 8 replies
- 6 kudos
- 6 kudos
Sorry I can't figure this out, the link you've added is irrelevant for passthrough credentials, if we add it the cluster won't be passthrough, Is there a way to add this just for a specific folder? while keeping passthrough for the rest?
- 6 kudos
- 1296 Views
- 0 replies
- 4 kudos
Databricks Office Hours Register for Office Hours to participate in a live Q&A session with Databricks experts! Our next event is scheduled for Ju...
Databricks Office HoursRegister for Office Hours to participate in a live Q&A session with Databricks experts! Our next event is scheduled for June 22nd from 8:00 am - 9:00am PT.This is your opportunity to connect directly with our experts to ask any...
- 1296 Views
- 0 replies
- 4 kudos
- 4991 Views
- 4 replies
- 2 kudos
Resolved! Unable to delete a DBFS folder
Hello everyone,I've created by error a DBFS folder named : ${env]But when I run this command :dbutils.fs.rm("/mnt/${env]")It returns me this error : java.net.URISyntaxException: Illegal character in path at index 12: /mnt/$%7Benv]How can I do please ...
- 4991 Views
- 4 replies
- 2 kudos
- 4623 Views
- 2 replies
- 3 kudos
Why does my Notebook fails if I try to load a function from another Notebook in Repos in Databricks
My function in func.pydef lower_events(df): return df.withColumn("event",f.lower(f.col("event")))My main notebook import pyspark.sql.functions as f from pyspark.sql.functions import udf, col, lower import sys sys.path.append("..") from folder.func...
- 4623 Views
- 2 replies
- 3 kudos
- 3 kudos
@Kaniz Fatma https://community.databricks.com/s/question/0D58Y00008ouo6xSAA/how-to-fetch-environmental-variables-saved-in-one-notebook-into-another-notebook-in-databricks-repos-and-notebooksCan you please look into this
- 3 kudos
- 3733 Views
- 2 replies
- 2 kudos
Resolved! Importing python function with spark.read.jdbc in to Repos
Hi all!Before we used Databricks Repos we used the run magic to run various utility python functions from one notebook inside other notebooks, fex like reading from a jdbc connections. We now plan to switch to repos to utilize the fantastic CI/CD pos...
- 3733 Views
- 2 replies
- 2 kudos
- 2 kudos
Thats...odd. I was sure I had tried that, but now it works somehow. I guess it has to be that now I did it with double quotation marks. Thanks anyway! Works like a charm.
- 2 kudos
- 8820 Views
- 7 replies
- 2 kudos
Read temporary views in SQL Analytics
I'm having issues trying to read temporary views in SQL Analytics module. Ive managed to create temporary views based on a query but I don't know how to read from them? Just using the name of the view returns "Table or view not found".
- 8820 Views
- 7 replies
- 2 kudos
- 2 kudos
No, I'm actually having issues reading from the view in the same session that created it. Using the same view name I get a table or view not found.
- 2 kudos
- 2067 Views
- 1 replies
- 1 kudos
Autoloader event vs directory ingestion
For a production work load containing around 15k gzip compressed json files per hour all in a YYYY/MM/DD/HH/id/timestamp.json.gz directoryWhat would be the better approach on ingesting this into a delta table in terms of not only the incremental load...
- 2067 Views
- 1 replies
- 1 kudos
- 1 kudos
@Kaniz Fatma So i've not found a fix for the small file problem using autoloader, seems to struggle really badly against large directories, had a cluster running for 8h stuck on "listing directory" part with no end, cluster seemed completely idle to...
- 1 kudos
- 6764 Views
- 4 replies
- 6 kudos
Unable to import .dbc files in Databricks for "Databricks Developer Foundation Capstone"
Hi,I am not able to import .dbc file into Databricks workspace for "Databricks Developer Foundation Capstone". When I click import the error message is displayed. Secondly when I click the github link in the 'Download the capstone' error 404 is displ...
- 6764 Views
- 4 replies
- 6 kudos
- 6 kudos
Hello @Karthikeyan.r3@cognizant.com R I agreed with Hubert. please write to https://help.databricks.com
- 6 kudos
- 5059 Views
- 1 replies
- 1 kudos
Resolved! Can we reuse checkpoints in Spark Streaming?
I am reading data from a Kafka topic, say topic_a. I have an application, app_one which uses Spark Streaming to read data from topic_a. I have a checkpoint location, loc_a to store the checkpoint. Now, app_one has read data till offset 90.Can I creat...
- 5059 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @John Constantine,Is not recommended to share the checkpoint with your queries. Every streaming query should have their own checkpoint. If you can to start at the offset 90 in another query, then you can define it when starting your job. You can ...
- 1 kudos
- 2017 Views
- 1 replies
- 1 kudos
- 2017 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi,Could you share more details on what you have tried? please provide more details.
- 1 kudos
- 9770 Views
- 5 replies
- 9 kudos
Visualisation libraries does not work on Databricks
Why is it that certain Python visualisation libraries do not work on Databricks? I am trying to install (via pip) and work with some data visualisation libraries - they work perfectly in a normal Jupyter Notebook but not on a Databricks notebook envi...
- 9770 Views
- 5 replies
- 9 kudos
- 20444 Views
- 13 replies
- 6 kudos
Resolved! Did any one in the community create Permanent Functions using using python script ? I tried but i am getting the below error, Please advise
Hi Team, When i am trying to register a permanant function i am getting the below error.%sqlCREATE FUNCTION simple_udf AS 'SimpleUdf' USING JAR '/tmp/SimpleUdf.jar';%sqlselect simple_udf(2)Error Details : com.databricks.backend.common.rpc.Databricks...
- 20444 Views
- 13 replies
- 6 kudos
- 6 kudos
hi @Werner Stinckens @Jose Gonzalez @Hubert Dudek @Kaniz Fatma Thanks for all the help, Appreciate it. I was able to create permanent functions and use eclipse to create the runnable jar. However, Does anyone have any idea on how to deploy t...
- 6 kudos
| User | Count |
|---|---|
| 1616 | |
| 786 | |
| 367 | |
| 349 | |
| 287 |
