- 1587 Views
- 1 replies
- 0 kudos
Hi , I am trying to schedule - Exam: Databricks Certified Associate Developer for Apache Spark 3.0 - Python.In the cart --> I couldn't proceed ...
Hi , I am trying to schedule - Exam: Databricks Certified Associate Developer for Apache Spark 3.0 - Python.In the cart --> I couldn't proceed without entering voucher. I do not have voucher.Please help
- 1587 Views
- 1 replies
- 0 kudos
- 0 kudos
Can someone Please respond to my above question ? Can i write certification test without Voucher ?
- 0 kudos
- 16072 Views
- 3 replies
- 4 kudos
Resolved! How to convert lat/long to geohash in databricks using geohashTools R library
I continues to receive a parsing error when attempting to convert lat/long data to a geohash in data bricks . I've tried two coding methods in R and get the same error.library(geohashTools)Method #1my_tbl$geo_hash <- gh_encode(my_tbl$Latitude, my_tbl...
- 16072 Views
- 3 replies
- 4 kudos
- 4 kudos
The problem was I was trying to run the gh_encode function on a Spark dataframe. I needed to collect the date into a R dataframe then run the function.
- 4 kudos
- 21634 Views
- 3 replies
- 7 kudos
Resolved! How to set retention period for a delta table lower than the default period? Is it even possible?
I am trying to set retention period for a delta by using following commands.deltaTable = DeltaTable.forPath(spark,delta_path)spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")deltaTable.logRetentionDuration = "interval 1...
- 21634 Views
- 3 replies
- 7 kudos
- 7 kudos
There are two ways:1) Please set in cluster (Clusters -> edit -> Spark -> Spark config):spark.databricks.delta.retentionDurationCheck.enabled false 2) or just before DeltaTable.forPath set (I think you need to change order in your code):spark.conf.se...
- 7 kudos
- 6817 Views
- 5 replies
- 12 kudos
Resolved! Query delta tables using databricks cluster in near real time.
I'm trying to query delta tables using JDBC connector in a Ruby app. I've noticed that it takes around 8 seconds just to connect with databricks cluster and then additional time to run the query.The app is connected to a web portal where users genera...
- 6817 Views
- 5 replies
- 12 kudos
- 12 kudos
Hi @Aman Sehgal Could you please check SQL endpoints? SQL endpoint uses a photon engine. It can reduce the query processing time. And Serverless SQL endpoint can accelerate the launch timemore info: https://docs.databricks.com/sql/admin/sql-endpoin...
- 12 kudos
- 4070 Views
- 1 replies
- 3 kudos
Resolved! Databricks - Jobs Last run date
Is there a way to get the last run date of job(s) ? I am trying to compile a report and trying to see if this output exists either in databricks jobs cli output or via api?
- 4070 Views
- 1 replies
- 3 kudos
- 3 kudos
Sure. Using Databricks jobs API you can get this information.Use the following API endpoint to get list of all the jobs and their executions till date in descending order.You can pass job_id as parameter to get runs of a specific job.https://<databri...
- 3 kudos
- 1325 Views
- 0 replies
- 3 kudos
March Madness + Data Here at Databricks we like to use (you guessed it) data in our daily lives. Today kicks off a series called Databrags �� ...
March Madness + Data Here at Databricks we like to use (you guessed it) data in our daily lives. Today kicks off a series called Databrags Databrags are glimpses into how Bricksters and community folks like you use data to solve everyday problems, e...
- 1325 Views
- 0 replies
- 3 kudos
- 3027 Views
- 1 replies
- 1 kudos
Resolved! Create data bricks service account
Hi all, I need to create service account users who can only query some delta tables. I guess I do that by creating the user and granting select right to the desired tables. But Data bricks requests a mail account for these users. Is there a way to cr...
- 3027 Views
- 1 replies
- 1 kudos
- 1 kudos
HI @Kaniz Fatma , I've checked the link but the standard method requires a mailbox and the user creation using SCIM API looks too complicated. I solved the issue, I created a mailbox for the service account and I created the user using that mailbox....
- 1 kudos
- 7766 Views
- 5 replies
- 1 kudos
Resolved! Problems with HiveMetastoreClient and internal Databricks Metastore.
I've been trying to use the HiveMetastoreClient class in Scala to extract some metadata from Databricks internal Metastore, without success. I'm currently using the 7.3 LTS runtime.The error seems to be related to some kind of inconsistency between...
- 7766 Views
- 5 replies
- 1 kudos
- 1 kudos
Thanks for the reference, @Atanu Sarkar .Seems a little odd to me that I'd need to change the internal Databricks Metastore table to add a column expected by the client default Scala client. I'm afraid this could cause issues with other users/jobs ...
- 1 kudos
- 10012 Views
- 4 replies
- 0 kudos
Resolved! If two Data Factory pipelines are run at the same time or share a window of execution do they share the Databricks spark cluster(if both have the same linked service)? ( job clusters are those that are create on the go, defined in the linked service).
Continuing the above case, does that mean if i have several like 5 ADF pipelines scheduled regularly at the same time, its better to use an existing cluster as all of the ADF pipelines would share the same cluster and hence the cost will be lower?
- 10012 Views
- 4 replies
- 0 kudos
- 0 kudos
for adf or job run we always prefer job cluster. but for streaming, you may consider using interactive cluster . but anyway you need to monitor the cluster load, if loads are high there will be chance to job slowness as well as failure. also data siz...
- 0 kudos
- 6311 Views
- 2 replies
- 1 kudos
Resolved! Correct setup and format for calling REST API for image classification
I trained a basic image classification model on MNIST using Tensorflow, logging the experiment run with MLflow.Model: "my_sequential" _________________________________________________________________ Layer (type) Output Shape ...
- 6311 Views
- 2 replies
- 1 kudos
- 1 kudos
@Anthony Gibbons may be this git should work with your use case - https://github.com/mlflow/mlflow/issues/1661
- 1 kudos
- 5460 Views
- 2 replies
- 1 kudos
Resolved! S3 sync from bucket to a mounted bucket causing a "[Errno 95] Operation not supported" error for some but not all files
Trying to sync one folder from an external s3 bucket to a folder on a mounted S3 bucket and running some simple code on databricks to accomplish this. Data is a bunch of CSVs and PSVs.The only problem is some of the files are giving this error that t...
- 5460 Views
- 2 replies
- 1 kudos
- 1 kudos
@Matthew Tribby does above suggestion work. Please let us know if you need further help on this. Thanks.
- 1 kudos
- 9947 Views
- 3 replies
- 3 kudos
Resolved! AttributeError: module 'dbutils' has no attribute 'fs'
Hi,Using db in SageMaker to connect EC2 to S3. Following other examples I get 'AttributeError: module 'dbutils' has no attribute 'fs'....I guess Im missing an import?
- 9947 Views
- 3 replies
- 3 kudos
- 3 kudos
agree with @Werner Stinckens . also may try importing dbutils - @ben Hamilton
- 3 kudos
- 3868 Views
- 3 replies
- 5 kudos
Resolved! Understand Spark DataFrames verse R DataFrames
CommunityI’ve been struggling with utilizing R language in databricks and after reading “Mastering Spark with R,” I believe my initial problems stemmed from not understating the difference between Spark DataFrames and R DataFrames within the databric...
- 3868 Views
- 3 replies
- 5 kudos
- 5 kudos
As Spark dataframes are handled in distributed way on workers it is better just to use Spark dataframes. Additionally collect is executed on driver and takes whole dataset into memory so it is shouldn't be used in production.
- 5 kudos
- 6103 Views
- 3 replies
- 6 kudos
Resolved! Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?
We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?
- 6103 Views
- 3 replies
- 6 kudos
- 6 kudos
Yes it is possible. Additionally mount is permanent and done in dbfs so it is enough to run it one time. you can have for example following configuration:In Azure you can have 2 databricks workspace,cluster in every workspace can have env variable is...
- 6 kudos
- 9562 Views
- 5 replies
- 1 kudos
Resolved! Quick way to know delta table is zordered
Hello,I created a delta table table using SQL and specifying the partitioning and zorder strategy. I then loaded data into it for the first time by doing a write as delta with mode of append and save as table. However, I don’t know of a way to verify...
- 9562 Views
- 5 replies
- 1 kudos
- 1 kudos
If there is no data then lines 10 and 11 will not have any impact. I am assuming that line (1-5) is creating an empty table but the actual load is happening when you do df.write operation. Also delta.autoOptimize.autoCompact will not trigger the z-or...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
