- 8562 Views
- 3 replies
- 0 kudos
Error when calling SparkR from within a Python notebook
I’ve had success with R magic (R cells in a Python notebook) and running an R script from a Python notebook, up to the point of connecting R to a Spark cluster. In either case, I can’t get a `SparkSession` to initialize. 2-cell (Python) notebook exa...
- 8562 Views
- 3 replies
- 0 kudos
- 0 kudos
The answer I can give you to have this work for you is to call the R notebooks from your Python notebook. Just save each dataframe as a delta table to pass between the languages.How to call a notebook from another notebook? here is a link
- 0 kudos
- 3730 Views
- 2 replies
- 2 kudos
Using dbutils.fs.ls on URI with square brackets results in error
Square brackets in ADLS are accepted, so why can't I list the files in the folder? I have tried escaping the square brackets manually, but then the escaped values are re-escaped from %5B to %255B and %5D to %255D. I get:URISyntaxException: Illegal ...
- 3730 Views
- 2 replies
- 2 kudos
- 2 kudos
@Joshua Stafford​ :The URISyntaxException error you are encountering is likely due to the fact that square brackets are reserved characters in URIs (Uniform Resource Identifiers) and need to be properly encoded when used in a URL. In this case, it ap...
- 2 kudos
- 9604 Views
- 11 replies
- 4 kudos
Certificate and Badge not received
I have cleared my certification exam, Databricks Certified Data Engineer Associate on 07 April 2023. I haven't received any certification or badge yet.I sent an email to training databricks and they told me that this problem has been solved...Any Hel...
- 9604 Views
- 11 replies
- 4 kudos
- 4 kudos
Hello all,We are going through a contract renewal with our vendor, Accredible. Once our new contract goes through you will get your badge this week. Thank you for understanding.
- 4 kudos
- 48000 Views
- 13 replies
- 8 kudos
Resolved! Not able to generate Access Token for Service Principal using rest API
I am trying to generate a Databricks token for a service principal (SP). I have created the SP in Azure AD and have used the Databricks rest api to add it as an admin.When using the Databricks rest API "/api/2.0/token-management/on-behalf-of/tokens" ...
- 48000 Views
- 13 replies
- 8 kudos
- 8 kudos
Hi all,I believe I found a temporary fix for this -Generate an AAD token for the service principle in Azure. Follow this guide if you don't know how to -https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-toke...
- 8 kudos
- 7648 Views
- 2 replies
- 0 kudos
Unexpected failure while fetching notebook - What can we do from our side?
Hello!We got some pipelines running in Azure Data Factory that call Databricks Notebooks to run data transformations. This morning at 6:21 AM (UTC) we got an error " Unexpected failure while fetching notebook" inside a notebook that calls another one...
- 7648 Views
- 2 replies
- 0 kudos
- 0 kudos
@Rita Fernandes​ :Based on the error message you provided, it seems like the issue might be related to the version mismatch between the ANTLR tool used for code generation and the current runtime version. Additionally, the error message suggests that...
- 0 kudos
- 3221 Views
- 3 replies
- 0 kudos
map_keys() returns an empty array in Delta Live Table pipeline.
We are exploding a map type column into multiple columns based on the keys of the map column. Part of this process is to extract the keys of a map type column called json_map as illustrated in the snippet below. The code executes as expected when run...
- 3221 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Suteja Kanuri​ , Thank you for you response and explanation. The code I have shown above is not the exact snippet we are using. Please find the exact snippet below. We are dynamically extracting the keys of the map and then using getitem() to mak...
- 0 kudos
- 5083 Views
- 0 replies
- 0 kudos
Types of RAM: An In-Depth OverviewRandom Access Memory (RAM) is an essential component of any computer system, responsible for temporarily storing dat...
Types of RAM: An In-Depth OverviewRandom Access Memory (RAM) is an essential component of any computer system, responsible for temporarily storing data that the CPU (Central Processing Unit) needs to access quickly. It allows for faster data retrieva...
- 5083 Views
- 0 replies
- 0 kudos
- 17096 Views
- 3 replies
- 0 kudos
Cannot import name 'Test' from partially initialized module 'databricks_test_helper'
I have done install, with this command ' pip install databricks_test_helper 'next get ImportError messages when i try running this code on cloud databricks ;from databricks_test_helper import *expected = set([(s, 'double') for s in ('AP', 'AT', 'PE'...
- 17096 Views
- 3 replies
- 0 kudos
- 0 kudos
@Burhanudin Badiuzaman​ :The error message suggests that there may be a circular import happening within the databricks_test_helper module, which is preventing the Test class from being properly imported.One possible solution is to import the Test cl...
- 0 kudos
- 8833 Views
- 11 replies
- 2 kudos
Resolved! DBT Job Type Authenticating to Azure Devops for git_source
we are trying to execute the databricks jobs for dbt task type but it is failing to autheticate to git. Problem is job is created using service principal but service principal don't seem to have access to the repo. few questions we have:1) can we giv...
- 8833 Views
- 11 replies
- 2 kudos
- 2 kudos
Hi @Rahul Samant​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest p...
- 2 kudos
- 12087 Views
- 7 replies
- 3 kudos
Authenticate Databricks REST API and access delta tables from external web service.
Hi All,We do have a requirement to access delta tables from external web service(Web UI). Presently we have tested it through jdbc connection and authenticated using PAT:Ex. jdbc:spark://[DATABRICKS_HOST]:443/default;transportMode=http;ssl=1;httpPath...
- 12087 Views
- 7 replies
- 3 kudos
- 7620 Views
- 0 replies
- 0 kudos
As companies grow and evolve, a Chief Technology Officer (CTO) becomes crucial in shaping the organization's technical direction and driving innov...
As companies grow and evolve, a Chief Technology Officer (CTO) becomes crucial in shaping the organization's technical direction and driving innovation. Regarding filling this critical leadership position, companies decide to either promote an existi...
- 7620 Views
- 0 replies
- 0 kudos
- 13839 Views
- 5 replies
- 0 kudos
Resolved! Getting date out of year and week
Hi all,I'm trying to get a date out of the columns year and week. The week format is not recognized. df_loaded = df_loaded.withColumn("week_year", F.concat(F.lit("3"),F.col('Week'), F.col('Jaar')))df_loaded = df_loaded.withColumn("date", F.to_date(F...
- 13839 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Pien Derkx​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 0 kudos
- 13270 Views
- 2 replies
- 0 kudos
How do I search for all the columns/field names starting with "XYZ"
I would like to do a big search on all field/columns names that contain "XYZ".I tried below sql but it's giving me an error.SELECT table_name,column_nameFROM information_schema.columnsWHERE column_name like '%<account>%'order by table_name, column_na...
- 13270 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Ian Fox​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your ...
- 0 kudos
- 2183 Views
- 2 replies
- 0 kudos
cannot install RMySQL "there is no package called ‘RMySQL’
cannot install RMySQL on databricks. i tried:install.packages("RMySQL")i got the error:Installing package into ‘/local_disk0/.ephemeral_nfs/envs/rEnv-c677bc4c-e6a3-40df-a5ab-bfd5d277e0c0’ (as ‘lib’ is unspecified) Warning: unable to access index for ...
- 2183 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @miru miro​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 0 kudos
- 10007 Views
- 4 replies
- 0 kudos
Difference between Databricks and local pyspark split.
I have noticed some inconsistent behavior between calling the 'split' fuction on databricks and on my local installation.Running it in a databricks notebook givesspark.sql("SELECT split('abc', ''), size(split('abc',''))").show()So the string is split...
- 10007 Views
- 4 replies
- 0 kudos
- 0 kudos
@Ivo Merchiers​ :The behavior you are seeing is likely due to differences in the underlying version of Apache Spark between your local installation and Databricks. split() is a function provided by Spark's SQL functions, and different versions of Spa...
- 0 kudos
| User | Count |
|---|---|
| 1632 | |
| 791 | |
| 513 | |
| 349 | |
| 287 |
