- 3696 Views
- 3 replies
- 3 kudos
Resolved! AWS Instance Profiles and DLT Pipelines
Hey everyone! I'm building a DLT pipeline that reads files from S3 (or tries to) and then writes them into different directories in my s3 bucket. The problem is I usually access S3 with an instance profile attached to a cluster, but DLT does not give...
- 3696 Views
- 3 replies
- 3 kudos
- 3 kudos
{ "clusters": [ { "label": "default", "aws_attributes": { "instance_profile_arn": "arn:aws:..." } }, { "label": "maintenance", "aws_attributes": { "instance_profile_arn": "arn:aws:..." ...
- 3 kudos
- 6426 Views
- 3 replies
- 5 kudos
How do you connect a folder path from your desktop to DB notebook?
I have a folder with multiples excel files that contains information from different cost centers, these files get update every week , im trying to upload all these files to the DB notebook , is there a way to connect the path directly to the DBFS to...
- 6426 Views
- 3 replies
- 5 kudos
- 5 kudos
Hello, Thanks for your question.You can mount a cloud object storage to dbfs and use them in a notebook. Please refer here.It is not possible to mount a local folder from desktop to dbfs. But you should be able to use the Databricks CLI to copy the e...
- 5 kudos
- 4194 Views
- 3 replies
- 6 kudos
- 4194 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Stephanie Rivera​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 6 kudos
- 21212 Views
- 2 replies
- 3 kudos
PySpark column object not callable using "when otherwise" transformation
The very first "when" function results in the posted error message (see image). The print statement of the count of df_td_amm works. A printSchema of the "df_td_amm" data frame confirms that "AGE" is a column. A select statement is also successful, s...
- 21212 Views
- 2 replies
- 3 kudos
- 3 kudos
the syntax is when(....).otherwise(...), not other(...)And there are some backslashes missing.
- 3 kudos
- 1299 Views
- 0 replies
- 0 kudos
Databricks migration from 7.3 LTS to X.x
We are currently on 7.3 LTS version with Python 3.7. I see that we are several versions off the latest 11.1 release https://docs.databricks.com/release-notes/runtime/releases.html.I see that the End of support for 11.1 is earlier than 10.4 LTS. What ...
- 1299 Views
- 0 replies
- 0 kudos
- 5888 Views
- 5 replies
- 1 kudos
Resolved! Jobs REST Api - Run a Job that is connected to a git_source
On Jobs REST API is possible to create a new Job, specifying a git_source.My question is about triggering the job.Still on Jobs REST Api is possible to trigger a job using the job_id, but I don't find a way to tell anyhow to Databricks, what's the en...
- 5888 Views
- 5 replies
- 1 kudos
- 1 kudos
Ah. Got it. So is your issue resolved or are you looking for further information.
- 1 kudos
- 3043 Views
- 0 replies
- 3 kudos
How to save json data to Delta Table: ParseError on Insert
I'm trying to save the returned json data from a requests API call to a Delta Table. I get a ParseError when I INSERT the response object which is in json format. The error shows the json data and a marker that states a ' or } or ) is missing. I v...
- 3043 Views
- 0 replies
- 3 kudos
- 7992 Views
- 7 replies
- 1 kudos
Resolved! Can't run a job that use GitHub as source
I have a list of jobs that are using the code in GitHub as source.Everything worked fine until yesterday. Yesterday, I noticed that all the job that were using GitHub as source were failing. Because of the following error: ``` Run result unavailable:...
- 7992 Views
- 7 replies
- 1 kudos
- 1 kudos
Just an update, to round this out. We investigated further internally, and found that although we have a cleanup process in place to remove the internal repos that are being checked out for workflows, it was failing to catch up due to the sheer volum...
- 1 kudos
- 4658 Views
- 2 replies
- 0 kudos
Jobs REST Api - Create new Job with a new Cluster, and install a Maven Library on the Cluster
I would need to use the Job REST API to create a Job on our databrick Cluster.At the Job Creation, is possible to specify an existing cluster, or, create a new one.I can forward alot of information to the Cluster, but what I would like to specify is ...
- 4658 Views
- 2 replies
- 0 kudos
- 0 kudos
@Antonio Davide Cali​ You can use the existing cluster in your json to use it for the job.To update or push libraries to the job, you can use the JobsUpdate API. As you want to push libraries to the cluster, you can push them using the new setting an...
- 0 kudos
- 2005 Views
- 0 replies
- 2 kudos
Cannot load spark-avro jars with databricksversion 10.4
Currently, I am facing an issue since the `databricks-connect` runtime on our cluster was updated to 10.4. Since then, I cannot load the jars for spark-avro anymore. By Running the following code from pyspark.sql import SparkSession spark = SparkSe...
- 2005 Views
- 0 replies
- 2 kudos
- 1072 Views
- 0 replies
- 5 kudos
www.databricks.com
New and Exciting! Databricks and Jupyter: Announcing ipywidgets in the Databricks NotebookBringing the interactivity of the Jupyter ecosystem into the LakehouseWe are excited to announce a deeper integration between the Databricks Notebook and the e...
- 1072 Views
- 0 replies
- 5 kudos
- 4861 Views
- 6 replies
- 4 kudos
Hello, I runn a notebook with datafactory and I have this error:Failure starting repl. Try detaching and re-attaching the notebook. But my notebook is...
Hello, I runn a notebook with datafactory and I have this error:Failure starting repl. Try detaching and re-attaching the notebook.But my notebook is attachingCan I attach notebook with a comand o r code ?
- 4861 Views
- 6 replies
- 4 kudos
- 4 kudos
Hey @oscar alvarez​ Thank you so much for getting back to us. It's really great of you to send in the solution. Would you be happy to mark it as best so that other members can find the solution more quickly?We really appreciate your time.Wish you a g...
- 4 kudos
- 1361 Views
- 0 replies
- 1 kudos
Integrating Databricks repos with Azure DevOps
Hi, Databricks community. I am trying to integrate Databricks shared folder notebooks with Azure DevOps GIT repositories. Can someone please point me to a basic training tutorial (or video) on how to get started and best practices?
- 1361 Views
- 0 replies
- 1 kudos
- 4238 Views
- 2 replies
- 0 kudos
Resolved! Cluster configuration / notebook panel
Hi,Is it possible to let regular users to see all running notebooks (in the notebook panel of the cluster) on a specific cluster they can use (attach and restart).by default admins can see all running notebooks and users can see only their own notebo...
- 4238 Views
- 2 replies
- 0 kudos
- 0 kudos
hi @Philippe CRAVE​ a user can see a notebook only if they have permission to that notebook. Else they won't be able to see it. Unfortunately there is no possibility for a normal user to see the notebooks attached to a cluster if they do not have per...
- 0 kudos
- 2987 Views
- 0 replies
- 2 kudos
Data unloading to S3 bucket from Databricks.
Hi,I am completely new to the Databricks & have a task to unload the data from Databricks table to the S3 location using java/sql. Is this possible? If yes can you please help me?
- 2987 Views
- 0 replies
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
