cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
saniafatimi
by • New Contributor II
  • 3305 Views
  • 3 replies
  • 1 kudos

How to migrate power bi reports to databricks

I have a sample set of power bi(.pbix) reports with all dropdowns, tables, filters etc. Now I would like to migrate this reports to data bricks. whatever visuals are created in power bi, I would like to create same in data bricks from scratch. I wou...

Data Engineering
Reply
  • 3305 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Neeljy
New Contributor II
  • 1 kudos

We must ensure you are sure that the Databricks cluster is operational. These are the steps needed for integration between Azure Databricks into Power BI Desktop.1. Constructing the URL for the connectionConnect to the cluster, and click the Advanced...

  • 1 kudos
2 More Replies
tj-cycyota
by • Databricks Employee
  • 2114 Views
  • 4 replies
  • 1 kudos

Can you use the Databricks API from a notebook?

I want to test out different APIs directly from a Databricks notebook instead of using Postman or CURL. Is this possible?

Data Engineering
Reply
  • 2114 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Boris2
New Contributor II
  • 1 kudos

@Panda There is no REST API for databricks. "RE" in REST stands for Ready Everywhere. You cannot connect to the API in workspace 1, from a notebook in workspace 2. Therefor it is Not Ready Everywhere. Workspace 1 cannot resolve the hostname for Works...

  • 1 kudos
3 More Replies
User16752245312
by • New Contributor III
  • 17072 Views
  • 3 replies
  • 3 kudos

How can I make Databricks API calls from notebook?

Access to Databricks APIs require the user to authenticate. This usually means creating a PAT (Personal Access Token) token. Conveniently, a token is readily available to you when you are using a Databricks notebook.databricksURL = dbutils.notebook....

Data Engineering
Reply
  • 17072 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Panda
Valued Contributor
  • 3 kudos

@User16752245312  You can use Databricks Secret Scope to manage sensitive data such as personal access tokens (PATs) securely. Storing your token in a secret scope ensures you don’t hard-code credentials in your notebook, making it more secure.For mo...

  • 3 kudos
2 More Replies
abhinandan084
by • New Contributor III
  • 21114 Views
  • 18 replies
  • 12 kudos

Community Edition signup issues

I am trying to sign up for the community edition (https://databricks.com/try-databricks) for use with a databricks academy course. However, I am unable to signup and I receive the following error (image attached). On going to login page (link in ora...

0693f000007OoQjAAK
Data Engineering
Reply
  • 21114 Views
  • 18 replies
  • 12 kudos
View Replies
Latest Reply
brokeTechBro
New Contributor II
  • 12 kudos

Hello,I get "An error occurred, try again"I am exhausted from trying... also from solving the puzzle to prove I'm not a robot

  • 12 kudos
17 More Replies
gtaspark
by • New Contributor II
  • 50376 Views
  • 9 replies
  • 5 kudos

Resolved! How to get the total directory size using dbutils

Is there a way to get the directory size in ADLS(gen2) using dbutils in databricks? If I run this dbutils.fs.ls("/mnt/abc/xyz") I get the file sizes inside the xyz folder( there are about 5000 files), I want to get the size of the XYZ folder how ca...

Data Engineering
Reply
  • 50376 Views
  • 9 replies
  • 5 kudos
View Replies
Latest Reply
User16788316720
New Contributor III
  • 5 kudos

File size is only specified for files. So, if you specify a directory as your source, you have to iterate through the directory. The below snippet should work (and should be faster than the other solutions).import glob   def get_directory_size_in_byt...

  • 5 kudos
8 More Replies
brickster_2018
by • Databricks Employee
  • 11737 Views
  • 3 replies
  • 6 kudos

Resolved! How to add I custom logging in Databricks

I want to add custom logs that redirect in the Spark driver logs. Can I use the existing logger classes to have my application logs or progress message in the Spark driver logs.

Data Engineering
Reply
  • 11737 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Kaizen
Valued Contributor
  • 6 kudos

1) Is it possible to save all the custom logging to its own file? Currently it is being logging with all other cluster logs (see image) 2) Also Databricks it seems like a lot of blank files are also being created for this. Is this a bug? this include...

  • 6 kudos
2 More Replies
PHorniak
by • New Contributor II
  • 15914 Views
  • 3 replies
  • 4 kudos

Resolved! AttributeError: 'DataFrame' object has no attribute 'rename'

Hello, I am doing the Data Science and Machine Learning course. The Boston housing has unintuitive column names. I want to rename them, e.g. so 'zn' becomes 'Zoning'. When I run this command: df_bostonLegible = df_boston.rename({'zn':'Zoning'}, axi...

Data Engineering
Reply
  • 15914 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
KrunalLathiya
New Contributor II
  • 4 kudos

If df_boston is a DataFrame, but you still face issues, try an alternative syntax: df_boston = df_boston.rename(columns={'zn': 'Zoning'}).Make sure df_boston is a proper DataFrame and you're using a recent version of Pandas.

  • 4 kudos
2 More Replies
User16826994223
by • Honored Contributor III
  • 4044 Views
  • 3 replies
  • 2 kudos

TPC -DS test On databricks

If I want to run TPC-DS test on databricks what are the steps involved, do we have already daya available on databricks file system or I have to download or create from somewhere.

Data Engineering
Reply
  • 4044 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
aladda
Databricks Employee
  • 2 kudos

See the spark-sql-perf repo for details on how to run benchmark tests using TPC-DS - https://github.com/databricks/spark-sql-perf

  • 2 kudos
2 More Replies
tj-cycyota
by • Databricks Employee
  • 8171 Views
  • 2 replies
  • 1 kudos

Whats the difference between magic commands %pip and %sh pip

In Databricks you can do either %pipor %sh pipWhats the difference? Is there a recommended approach?

Data Engineering
Reply
  • 8171 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
stefnhuy
New Contributor III
  • 1 kudos

Hey there, User16776431030.Great question about those magic commands in Databricks! Let me shed some light on this mystical matter.The %pip and %sh pip commands may seem similar on the surface, but they're quite distinct in their powers. %sh pip is l...

  • 1 kudos
1 More Replies
User15986662700
by • New Contributor III
  • 4270 Views
  • 4 replies
  • 1 kudos

How to integrate Databricks and Spark to Secure HBase cluster with Kerberos?

Data Engineering
Reply
  • 4270 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
User15986662700
New Contributor III
  • 1 kudos

Yes, it is possible to connect databricks to a kerberized hbase cluster. The attached article explains the steps. It consists of setting up a kerberos client using a keytab in the cluster nodes, installing the hbase-spark integration library, and set...

  • 1 kudos
3 More Replies
Madman
by • New Contributor II
  • 13310 Views
  • 5 replies
  • 6 kudos

Snowflake connection to Databricks error

When I am trying to read snowflake table from my databricks notebook, it is giving the error as:df1.read.format("snowflake") \.options(**options) \.option("query", "select * from abc") \.save()Getting below errorjava.sql.SQLException: No suitable dri...

Data Engineering
Reply
  • 13310 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
pdiegop
New Contributor II
  • 6 kudos

@anurag2192 did you managed to solve it?

  • 6 kudos
4 More Replies
Anonymous
by • Not applicable
  • 5371 Views
  • 2 replies
  • 1 kudos

Resolved! If I know the workspaceID, how can I figure out the the respective Databricks workspace URL?

Data Engineering
Reply
  • 5371 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
wmespi
New Contributor II
  • 1 kudos

Is this random number not possible to extract from the notebook context? It is available in the browser_hash but that is not populated when running a job.Is this random number static or does it change over time? If it is static, it can then be hardco...

  • 1 kudos
1 More Replies
Aj2
by • New Contributor III
  • 9895 Views
  • 4 replies
  • 1 kudos

Resolved! How to connect to DB2-AS400?

What are the steps needed to connect to a DB2-AS400 source to pull data to lake using Databricks? I believe it requires establishing a jdbc connection, but I couldnot find much details online

Data Engineering
Reply
  • 9895 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Ajay Menon​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
3 More Replies
keenan_jones7
by • New Contributor II
  • 10875 Views
  • 2 replies
  • 5 kudos

Cannot create job through Jobs API

import requests import json instance_id = 'abcd.azuredatabricks.net' api_version = '/api/2.0' api_command = '/jobs/create' url = f"https://{instance_id}{api_version}{api_command}" headers = {'Authorization': 'Bearer myToken'} params = { "settings...

Data Engineering
Reply
  • 10875 Views
  • 2 replies
  • 5 kudos
View Replies
Latest Reply
rAlex
New Contributor III
  • 5 kudos

@keenan_jones7​ I had the same problem today. It looks like you've copied and pasted the JSON that Databricks displays in the GUI when you select View JSON from the dropdown menu when viewing a job.In order to use that JSON in a request to the Jobs ...

  • 5 kudos
1 More Replies
Labels