- 1298 Views
- 1 replies
- 0 kudos
New task orchestration tool
Hi everybody! Is it possible to use one job cluster for multiple tasks using new task orchestration tool? If so then please tell how. If it's impossible then this new tool looks useless Regards, Daniil.
- 1298 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks team, are there any news about it?
- 0 kudos
- 3252 Views
- 1 replies
- 1 kudos
Where is dbfs mounted with community edition?
The the regular version of databricks, the dbfs is mounted at /dbfs. This does not seem to be the case with community edition. I am seeking more details.
- 3252 Views
- 1 replies
- 1 kudos
- 1 kudos
If you are using DBR 7.x. or newer versions the dbfs is disabled. You have to use dbutils.fs.cp commands as a workaround.
- 1 kudos
- 2902 Views
- 0 replies
- 1 kudos
How to run databricks notebooks on databricks cluster from VSCODE?
I want to use the VS Code IDE to edit/write/work on jupyter notebooks but have the code run off of the databricks cluster. Does anyone know of a way for me to force the jupyter notebook server on VS Code to point towards the databricks cluster and e...
- 2902 Views
- 0 replies
- 1 kudos
- 3863 Views
- 1 replies
- 2 kudos
Resolved! EC2 instances are not stoped after the cluster terminated
I found that Databricks did not stop and delete ec2 instances of clusters. After the cluster terminate , the ec2 instances are still running.
- 3863 Views
- 1 replies
- 2 kudos
- 2 kudos
Please make sure the Databricks IAM role does have all required permission mentioned https://docs.databricks.com/administration-guide/account-api/iam-role.html#0-language-Your%C2%A0VPC,%C2%A0defaultMake sure you did not change the EC2 tags especially...
- 2 kudos
- 3583 Views
- 0 replies
- 0 kudos
Structured streaming from an overwrite delta path
Hi experts, I need to ingest data from an existing delta path to my own delta lake. The dataflow is as shown in the diagram: Data team reads full snapshot of a database table and overwrite to a delta path. This is done many times per day, but...
- 3583 Views
- 0 replies
- 0 kudos
- 1117 Views
- 0 replies
- 0 kudos
GroupBy in a multi node environment
I have a group of rows with Information on a nested product calls. example- Trxn1-product1-caller1-local1 Trxn1-Product1-local1-local2 Trxn1-Product1-local2-local3 here’s is a expected calls for a product product1-caller1-local1 Product1-local1-loc...
- 1117 Views
- 0 replies
- 0 kudos
- 1729 Views
- 1 replies
- 1 kudos
What is the equivalent command for constructing the filepath in Databricks on AWS? filepath = f"{working_dir}/keras_checkpoint_weights.ckpt"
dbutils.fs.mkdirs("/foobar/")See https://docs.databricks.com/data/databricks-file-system.html
- 1729 Views
- 1 replies
- 1 kudos
- 1 kudos
To access DBFS via local file APIs, you can try /dbfs/<foobar>https://docs.databricks.com/data/databricks-file-system.html?_ga=2.41953189.1820496821.1627689131-1247613683.1627514237#local-file-apis
- 1 kudos
- 865 Views
- 0 replies
- 0 kudos
Databricks-connect: is it safe to store my PAT in plaintext?
i am getting began with databricks-connect to hook up with my azure databricks cluster the use of a private get admission to token. it looks 192.168.1.254 like this token is being stored in a uncooked text report. if it is proper, is there a extra r...
- 865 Views
- 0 replies
- 0 kudos
- 1227 Views
- 0 replies
- 0 kudos
How can I connect to Databricks via a local IDE?
I want to run some unit tests on my code, but Databricks can't seem to handle running formal unit testing libraries due to the lack of command line. From Googling, it appears it's possible to run notebooks and such from IntelliJ if using Scala, rath...
- 1227 Views
- 0 replies
- 0 kudos
- 1080 Views
- 0 replies
- 1 kudos
Unknown token of a workspace user send login request to the workspace
From Audit logs, I found one workspace user keep sending tokenLogin with a token that I cannot identify. The user personal access tokens are all with different ID.
- 1080 Views
- 0 replies
- 1 kudos
- 17577 Views
- 3 replies
- 0 kudos
- 17577 Views
- 3 replies
- 0 kudos
- 0 kudos
@Ryan Chynoweth and @Sean Owen are both right, but I have a different perspective on this.Quick side note: you can also configure your cluster to execute with only a driver, and thus reducing the cost to the cheapest single VM available. In the cl...
- 0 kudos
- 3076 Views
- 3 replies
- 0 kudos
Resolved! Cluster Sizing
How big should my cluster be? How do I know how many nodes to use or the kind of instance to use?
- 3076 Views
- 3 replies
- 0 kudos
- 0 kudos
>How big should my cluster be? This would really depend on the use case. Some general guiding principles could be found here https://docs.databricks.com/clusters/cluster-config-best-practices.html
- 0 kudos
- 2717 Views
- 2 replies
- 0 kudos
Issue loading spark Scala library
We have a proprietary spark scala library, which is necessary for me to do my work. We build a release version once a week and store it in a specific s3 location (so the most up-to-date prod version is always stored in the same place). But so far I c...
- 2717 Views
- 2 replies
- 0 kudos
- 0 kudos
Databrick's curriculum team solved this problem by creating our own Maven repo and it's easier than it sounds. To do this, we took an S3 bucket, converted it to a public website, allowing for standard file downloads, and then within that bucket creat...
- 0 kudos
- 4433 Views
- 3 replies
- 0 kudos
Why does the display name of widgets not match the specified name in SQL?
However, I have no problem accessing the widget with the specified name.
- 4433 Views
- 3 replies
- 0 kudos
- 0 kudos
Yep, I figured out the issue now. Both of you gave the right information to solve the problem. My first mistake was as Jacob mentioned, `date` is actually a dataframe object here. To get the string date, I had to do similar to what Amine suggested. S...
- 0 kudos
- 4382 Views
- 2 replies
- 0 kudos
Resolved! Is there a way to validate the values of spark configs?
We can set for example:spark.conf.set('aaa.test.junk.config', 99999) , and then run spark.conf.get("aaa.test.junk.config”) which will return a value.The problem occurs when incorrectly setting to a similar matching property.spark.conf.set('spark.sql....
- 4382 Views
- 2 replies
- 0 kudos
- 0 kudos
You would solve this just like we solve this problem for all lose string references. Namely, that is to create a constant that represents the key-value you want to ensure doesn't get mistyped.Naturally, if you type it wrong the first time, it will be...
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 787 | |
| 384 | |
| 349 | |
| 287 |
